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ABSTRACT 

Computer  software  has  taken  an  increasingly  larger  role  in  the  U.S.  Navy.  It  is 
used  in  nearly  every  facet  of  naval  operations,  from  administrative  chores  to  controlling 
complex  weapons  systems.  Because  of  the  high  cost  of  software  and  the  potential  for 
inadvertent  misuse,  it  is  important  that  software  be  easy  to  use  and  understand.  This 
thesis  explores  the  methods  and  techniques  available  for  conducting  software  usability 
evaluations.  Using  what  described  in  this  thesis,  actual  software  usability  testing  is  done 
on  a  recently  developed  Web  site.  The  Web  site  [http://spitfire.avsafety.nps.navy.mil] 
evaluated  in  this  study  is  designed  to  allow  aviation  units  to  complete  a  safety  survey 
online.  This  thesis  describes  the  usability  test  conducted  on  the  Aviation  Command 
Safety  Assessment  (ACSA)  Web  site  and  establishes  a  methodology  that  can  be  used  on 
any  future  Navy  Web  site.  The  results  of  this  usability  test  show  that  improvement  can  be 
made  to  the  interface  design  and  presentation  of  material. 
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I.  INTRODUCTION 


A.  PURPOSE 

The  purpose  of  this  thesis  is  to  develop  and  apply  a  methodology  that  can  be  used 
to  assess  the  usability  of  a  Web-based  user  Interface  for  administering  the  Aviation 
Command  Safety  Assessment  (ACSA)  Survey  questionnaire  via  Internet  Technology. 
This  questionnaire  is  needed  to  support  the  School  of  Aviation  Safety  in  applying  a 
methodology  for  assessing  the  effectiveness  of  Naval  organizations  in  the  management  of 
risks  associated  with  flight  operations. 

In  April  1996,  Commander  Naval  Air  Pacific  chartered  an  Aviation  Safety  Quality 
Management  Board  (QMB)  to  explore  ways  in  which  the  Naval  aviation  mishap  rate 
might  be  lowered.  In  order  to  establish  a  baseline  analysis  of  the  “safety  climate”  at  the 
squadron  level.  Dr.  Anthony  Ciavarelli  was  appointed  as  the  principal  investigator  for  the 
Aviation  Safety  Command  Survey.  He  developed  a  command  safety  assessment 
questionnaire  that  was  distributed  to  69  squadrons  in  the  fall  of  1996.  The  results  of  this 
initial  survey  provided  valuable  insight  into  the  safety  climate  of  squadrons,  wings,  and 
Type  Commanders  and  Naval  aviation  as  a  whole.  The  process  for  completing  this 
survey  was  a  paper  questionnaire,  with  submissions  being  returned  on  a  form  that  could 
be  read  by  a  scanner.  Distributing,  administering,  collecting  and  then  analyzing  these 
surveys  was  an  expensive,  time  consuming  and  labor-intensive  effort. 

In  March  1999,  an  internet-based,  prototype  model  for  administering  this  survey 
using  new  technologies  that  allowed  automated  survey  submission  and  analysis  was 
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by  two  Naval  Postgraduate  students  (Held  and  Mingo,  1999).  The  results  of  their  thesis 
is  a  web  site  [http://spitefire.avsafety.nps.navy.mil]  that  adheres  to  a  three-tier 
client/server  architecture. 

Experience  has  shown  that  the  value  of  computer  systems,  including  web  sites  is 
directly  tied  to  their  usability.  Therefore,  this  thesis  will  develop  guidelines  for  building 
Web  sites  which  focus  on  building  a  human-computer-interface  that  satisfies  ease  of  use 
standards. 

B.  OBJECTIVE 

In  order  to  achieve  the  stated  purpose  of  this  thesis  three  specific  objectives  were 
accomplished.  First,  a  selected  review  of  the  related  literature  on  Web  site  design  and 
usability  evaluation  was  conducted.  This  literature  review  yielded  an  understanding  of 
the  most  effective  and  insightful  methods  for  usability  engineering  for  the  web.  Second, 
using  the  knowledge  from  the  literature  review,  a  methodology  was  developed  for 
usability  testing  and  applied  to  the  Aviation  Command  Safety  Assessment  (ACSA) 
Survey  Web  site.  In  addition  to  the  formal  investigation  of  usability,  input  from  aviation 
safety  officers  who  used  the  web  survey  was  solicited.  Finally,  the  results  of  the  usability 
evaluation  were  analyzed  and  summarized  in  order  to  capture  the  benefits  of  this  study  for 
future  web  development. 

C.  PROBLEM  STATEMENT 

As  previously  indicated.  Held  and  Mingo  spent  at  least  a  year  and  several 
thousand  dollars  in  development  and  implementation  of  the  Aviation  Command  Safety 
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Assessment  (ACSA)  Survey  Web  site.  This  effort  does  not  guarantee  that  a  successful 
Web  site  for  completing  the  survey  has  been  produced.  No  matter  how  good  the  Web  site 
looks,  if  the  intended  user  has  difficulty  using  the  Web  site,  then  the  value  is  minimized. 
Over  time,  common  flaws  associated  with  poor  human-computer  interface  (HCI)  design 
have  been  identified.  Some  of  the  most  prevalent  problems  involve  screen  presentation 
formats  and  navigation  between  various  screens  (Gillespie,  1998).  Poor  design  also 
results  from  a  failure,  on  the  part  of  the  HCI,  to  reflect  the  form  and  level  of  user 
guidance  considered  most  appropriate.  If  the  HCI  is  poorly  designed,  then  the  potential 
exists  for  the  ultimate  user  of  the  Web  site  to  reject  it  simply  because  it  is  too  difficult  to 
use.  To  avoid  this  possibility  it  is  important  to  determine  what,  if  any,  ease  of  use 
problems  may  exist  in  the  Aviation  Command  Safety  Assessment  (ACSA)  Survey  Web 
site  interface  design.  This  determination  relies  on  ascertaining  and  applying  a  method  of 
usability  evaluation  that  answer  such  questions  as: 

•  Are  the  instructions  easy  to  read  and  understand? 

•  Does  the  arrangement  of  information  on  the  screen  make  sense? 

•  Are  users  able  to  quickly  learn  how  to  complete  the  survey? 

•  Is  navigation  within  the  Web  site  easy  to  accomplish? 

•  Are  the  selection  of  colors,  icons  and  graphics  intuitive? 

By  investigating  these  and  other  questions,  this  study  provides  a  preliminary 
indication  of  the  success  or  failure  of  the  Aviation  Command  Safety  Assessment  (ACSA) 
Survey  Web  site  and  potential  improvements  that  may  be  made  to  the  program. 
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D.  SCOPE  AND  LIMITATIONS 


It  is  not  the  intent  of  this  research  to  provide  an  in-depth  study  of  the  Aviation 
Command  Safety  Assessment  (ACSA)  Survey  questionnaire,  or  the  actual  development 
of  the  Aviation  Command  Safety  Assessment  (ACSA)  Web  site.  Prior  studies, 
specifically  Ciavarelli  and  Figlock,  et.  al,  (1996),  contain  substantial  coverage  of  these 
subjects  and  web  site  development  has  been  completed  by  Held  and  Mingo  (1999). 
Furthermore,  the  limitations  and  computer  hardware  requirements  needed  to  administer 
and  access  the  web  site  are  well  documented  in  these  publications  and  will  not  be 
discussed  here.  The  focus  of  this  study  is  the  Aviation  Command  Safety  Assessment 
(ACSA)  Web  site  design,  or  more  specifically,  the  interface  provided  by  the  software  as  it 
is  run  on  a  typical  personal  computer. 

E.  THESIS  ORGANIZATION 

This  thesis  consists  of  five  chapters.  Chapter  II  provides  a  review  of  selected 
literature  pertinent  to  web  site  design,  software  usability  and  usability  testing.  Chapter  III 
covers  the  methodology  behind  the  data  collection  and  Chapter  IV,  the  analysis  of  the 
data  collected.  The  remaining  chapter.  Chapter  V,  provides  the  conclusion  and 
recommendations.  The  questionnaire,  interview  transcripts  and  other  items  significant  to 
this  study  are  attached  as  appendices. 
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II.  BACKGROUND 


Building  a  medium  or  large  World  Wide  Web  site,  whether  for  distribution  over 
the  Internet  or  over  an  Intranet,  can  and  should  be  viewed  as  a  major  software 
development  effort.  Applications  on  the  Web  range  from  having  users  complete  simple 
tasks,  such  as  filling  in  a  form,  to  more  complex  undertakings,  such  as  collaborating  on  a 
project. 


The  tools  available  for  creating  Web  pages  allow  anyone  to  design  a  Web 
application.  But  the  Web-unlike  Windows,  Macintosh,  and  Unix--has  no  standards  for 
designing  applications.  People  come  to  the  Web  with  expectations  based  on  using  the 
other  platforms,  and  these  expectations  make  up  their  conceptual  model  of  using  an 
application  on  the  Web. 

Frequently,  in  Web  applications  users  get  lost  because  the  model  of  navigation  on 
the  Web  differs  from  that  in  other  applications.  This  difference  causes  confusion  among 
users,  but  application  developers  can  use  a  few  techniques  to  help  people  successfully  use 
the  application. 

Once  Web  site  creation  is  seen  as  software  development,  it  becomes  natural  to 
apply  the  tools  and  methods  we  have  learned  in  past  projects.  The  life  cycle  of  Web 
creation  is  identical  to  that  of  traditional  software:  requirements  gathering,  analysis, 
design,  implementation,  testing,  and  deployment.  And,  just  as  traditional  software 
development  should  have  a  functionality  and  a  usability  component,  so  should  Web 
development  efforts. 
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A. 


WHAT  IS  USABILITY? 


Usability  can  be  defined  as  the  degree  to  which  a  given  piece  of  software  assists 
the  person  sitting  at  the  keyboard  to  accomplish  a  task,  as  opposed  to  becoming  an 
additional  impediment  to  such  accomplishment.  The  broad  goal  of  usable  systems  is 
often  assessed  using  several  criteria: 

•  Ease  of  learning. 

•  Retention  of  learning  over  time. 

•  Speed  of  task  completion. 

•  Error  rate. 

•  Subjective  user  satisfaction. 

Methodologies  for  building  usable  systems  have  been  introduced  and  refined  over 
the  past  fifteen  or  so  years  under  the  discipline  of  Human-Computer  Interaction  (HCI). 
HCI  principles  include  an  early  and  consistent  focus  on  end  users  and  their  tasks, 
empirical  measurements  of  system  usage,  and  iterative  development.  Much  effort  has 
been  put  into  exploring  cognitive  models  of  human  behavior  as  it  relates  to  computer 
usage,  and  developing  guidelines  for  screen  layout  and  system  dialogues.  These  are 
predictive  endeavors  whose  purpose  is  to  assist  the  software  developer  in  the  initial  task 
analysis  and  system  design. 

But,  just  as  comprehensive  functional  requirements  and  a  detailed  design 
document  do  not  by  themselves  guarantee  that  a  programmer’s  final  code  will  be  correct, 
so  up-front  usability  guidelines  do  not  by  themselves  guarantee  a  usable  end  product.  In 
both  cases  a  distinct  validation  process  is  required. 


Usability  engineering  techniques  can  create  more  usable  Web  sites  by  helping 
developers  focus  on  the  users  rather  than  the  technology.  Understanding  users’  needs  and 
the  tasks  they  are  trying  to  accomplish  is  steadily  becoming  more  important  than  adopting 
the  latest  whiz-bang  feature.  Of  course,  if  the  latest  technology  fulfills  a  need,  it  should  be 
used;  but  incorporating  this  technology  just  to  be  cool  is  becoming  less  and  less 
acceptable. 

Improving  usability  is  not  a  black  art:  it  can  be  accomplished  by  heeding  common 
sense  advice  and  applying  techniques  that  help  us  understand  and  meet  the  needs  of  users. 
A  good  introduction  to  this  topic  is  Jakob  Nielsen’s  Usability  Engineering  (Nielsen  1994). 
Applying  usability  engineering  to  the  Web  is  not  that  difficult,  although  the  Web  does 
introduce  a  few  interesting  problems--and  solutions.  Darrell  Sano’s  Designing  Large- 
Scale  Web  Sites  (Sano  1996)  discusses  many  Web  usability  issues,  such  as  limitations 
and  constraints  for  Web  publishing  and  structuring  the  information  space. 

What  follows  is  a  review  of  usability  techniques  and  methods,  seen  in  light  of  the 
constraints  and  tradeoffs  of  the  World  Wide  Web.  The  best  place  to  see  usability 
engineering  applied  to  the  Web  is  the  Sun-on-Net  project  (Nielsen  1997c),  where 
usability  testing,  iterative  design,  and  other  activities  improve  the  usability  of  Sun’s  Web 
site. 


B.  INTERFACE  DESIGN  CONSIDERATIONS 
1.  Reading  on  the  World  Wide  Web 

In  order  to  design  interfaces  geared  toward  Web  based  clients  one  must  first  study 
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how  users  read  from  that  medium.  According  to  the  Nielsen  Norman  Group,  “User 
experience”  encompasses  all  aspects  of  the  end-user’s  interaction  with  the  organization, 
its  services,  and  its  products. 

The  first  requirement  for  an  exemplary  user  experience  is  to  meet 
the  exact  needs  of  the  customer,  without  fuss  or  bother.  Next  comes 
simplicity  and  elegance  that  produce  products  that  are  a  joy  to  own,  a  joy 
to  use.  True  user  experience  goes  far  beyond  giving  customers  what  they 
say  they  want,  or  providing  checklist  features.  In  order  to  achieve  high- 
quality  user  experience  in  a  company’s  offerings  there  must  be  a  seamless 
merging  of  the  services  of  multiple  disciplines,  including  engineering, 
marketing, ^graphical  and  industrial  design,  and  interface  design.  (Nielsen 
Norman  Group  http://www.nngroup.com/about/ 
userexperience.html,  February  22, 1999). 

These  statements  can  easily  be  applied  to  any  government  scenario,  where  the  customer  is 
analogous  to  any  user  of  an  organization’s  Web  site  and  references  to  the  company  equate 
to  the  organization  using  a  Web  site  for  constructive  purposes. 

A  study  conducted  by  John  Morkes  and  Jakob  Nielsen  found  that  79  percent  of 
test  users  scanned  and  new  page  they  came  across,  with  only  16  percent  reading  word-for- 
word  (Nielsen,  1997a).  Four  possible  reasons  for  this  particular  behavior  are  (Nielsen, 
1997b): 

•  Reading  from  computer  screens  is  tiring  and  approximately  25  percent 
slower  than  reading  from  paper; 

•  The  Web  is  a  user-driven  medium,  i.e.  users  feel  they  must  move  on 
and  click  on  things; 

•  Each  page  has  to  compete  with  literally  millions  of  other  pages  for  the 
user’s  attention; 

•  Modem  life  is  hectic. 

According  to  research  by  Nielsen  and  others,  individuals  do  not  want  to  invest  too  much 
time  in  any  one  page  while  searching  for  information  on  the  Web  because  the  page  may 


8 


not  contain  the  needed  information.  Thus,  the  page  is  quickly  scanned  for  relevance  in  a 
quick  once  over  before  clicking  on  the  next  page. 

Other  important  issues  are  the  credibility  of  the  information,  since  there  is  no 
guarantee  about  who  has  provided  the  information  posted  to  the  Web  site.  Studies  by 
Nielsen  have  shown  that  by  providing  hypertext  links  to  external  related  sites,  using  high 
quality  graphics,  and  employment  of  good  writing  practices,  credibility  or  perceived 
credibility  of  the  information  on  a  site  can  be  significantly  increased. 

2.  Writing  for  the  World  Wide  Web 

By  understanding  how  users  interface  with  and  read  on  the  Web,  insight  can  be 
grained  into  methods  of  effective  writing  for  the  Web.  Information  should  be  presented 
in  a  concise  and  intuitive  format  (Gahran,  1998a).  Keep  pieces  short,  by  breaking  up  if 
plausible  and  following  these  guidelines  (Gahran,  1998b ): 

•  Pare  down  whatever  you  are  saying  to  the  greatest  extent  that  is 
appropriate.  Consider  using  bulleted  lists  and  highlighted  key 
words. 

•  Make  sure  every  bit  of  content  has  a  point  that  will  matter  to 
the  user,  and  make  those  points  obvious. 

•  Break  up  longer  stretches  of  text  into  manageable  “chunks”. 

•  Attempt  to  predict  and  answer  all  the  major  natural  questions 
that  your  users  will  probably  have  about  your  organization  or 
your  site’s  content.  Consider  dropping  any  content  that  does 
not  focus  on  what  the  user  probably  wants  to  know. 

Writing  for  the  Web  is  more  akin  to  newspaper  journalism  than  to  any  other  form 
of  distributing  information,  including  marketing  publications,  magazine  articles,  books  or 
theses.  In  the  newspaper  format,  the  summary  or  conclusions  of  the  story  are  at  the  front 
of  the  story.  This  is  done  in  order  to  immediately  capture  and  retain  the  reader’s  attention. 
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If  the  reader  these  lines  and  is  interest  in  the  topic,  they  will  read  the  entire  article  and  be 
able  to  retrieve  details.  This  is  done  in  order  to  immediately  capture  and  retain  the 
reader’s  attention.  If  the  reader  reads  these  lines  and  is  interested  in  the  topic,  they  will 
read  the  entire  article  to  retrieve  details.  This  is  especially  true  for  the  front  page  in  the 
area  known  as  “above  the  fold.”  This  prime  real  estate  must  be  used  to  full  advantage  to 
make  the  sale  by  hooking  the  potential  customer  with  interesting  facts  in  order  to  entice 
them  to  purchase  the  paper.  In  the  same  vein,  each  Web  page  on  a  site  must  make  promise 
of  the  information  contained  therein,  as  well  as  in  the  lower  “drill-down”  layers.  The  most 
sought  after  information  should  be  summarized  or  encapsulated  at  the  top  of  the  page,  in 
the  writing  style  described  above.  The  user  will  scan  this  page  and  determine  if  it  contains 
the  information  needed,  and  read  further  or  drill-down  for  more  details.  Nielsen  (1996a) 
refers  to  this  type  of  writing  as  the  inverted  pyramid  style.  This  allows  the  page  creator  to 
identify  the  purpose  of  the  site  near  the  top  of  the  page  so  the  reader  can  easily  see  if  they 
want  to  invest  time  at  that  site. 

Creating  hyperlinks  in  text  serves  to  highlight  that  text  and  to  provide  a  place  so 
the  user  can  dill-down  to  obtain  more  information,  as  desired.  This  is  one  of  the  most 
powerful  features  of  the  Web,  and  this  is  what  creates  the  “webbing”  effect  of  the  Internet. 
This  allows  Web  writers  to  split  their  writing  into  smaller,  coherent  pieces  to  avoid  long 
scrolling  pages  (Nielsen,  1996b). 

C.  WEB  USABILITY  ENGINEERING  LIFE  CYCLE 

The  usability  engineering  life  cycle  (see  the  following  list  Nielsen  1994) 
represents  a  menu  of  choices  that  can  be  worked  into  the  broader  development  context  in 
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order  to  increase  usability.  The  techniques  focus  on  understanding  users,  promoting 
usable  designs,  finding  usability  problems,  and  understanding  other  constraints  in 
development. 

1.  Know  the  user. 

2.  Competitive  analysis. 

3.  Setting  usability  goals. 

4.  Parallel  design. 

5.  Participatory  design. 

6.  Coordinated  design  of  the  total  interface. 

7.  Applying  guidelines  and  heuristic  analysis. 

8.  Prototyping. 

9.  Empirical  testing. 

10.  Iterative  design. 

1 1.  Collecting  feedback  from  field  use. 

Steps  1, 7, 10  and  11  (Know  the  user.  Applying  guidelines  and  heuristic  analysis,  Iterative 
design,  and  Collecting  feedback  from  field  use)  are  most  pertinent  to  Web  usability  and 
are  elaborated  upon  in  the  sections  that  follow. 

1.  Know  the  User 

The  most  basic  of  usability  guidelines,  "know  the  user,"  can  be  very  difficult  on 
the  Web,  where  international  access,  exponential  growth,  and  wide-ranging  demographics 
(from  school-age  children  to  elderly  citizens)  are  the  norm.  It  is  also  difficult  to  understand 
what  people  really  want  from  this  new  medium,  in  which  so  much  is  possible.  The  many 
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surveys  of  general  Internet  demographics  will  provide  some  information  on  "who  is  out 
there,"  but  talking  to  users  is  still  the  best  way  to  get  a  handle  on  user-specific  usability 
problems. 

On  the  Web,  "know  the  user"  also  means  knowing  the  speed  of  their  Internet 
connection,  which  browser  version  they  are  using,  what  plug-ins  they  have,  and  so  on. 
Understanding  these  factors  plays  a  big  part  in  choosing  advanced  Web  technologies  and 
determining  when  they  can  be  safely  introduced. 

2.  Applying  Guidelines  and  Heuristic  Analysis 

Many  usability  problems  can  be  avoided  by  following  the  published  guidelines 
(Nielsen,  1996c)  and  by  adhering  to  some  general  rules  of  thumb  (see  the  Heuristics 
section  below).  Since  the  guidelines  and  heuristics  are  very  general,  additional  usability 
techniques,  such  as  empirical  testing,  are  needed  to  determine  the  best  solution  for  each 
particular  problem. 

3.  Iterative  Design 

Iterative  design  involves  a  four-step  process: 

•  Creating  something. 

•  Testing  it  out  with  users. 

•  Understand  the  strengths  and  weaknesses. 

•  Designing  a  better  version. 

This  fits  very  well  into  the  Web  "culture,"  in  which  a  site  does  not  have  to  be 
perfect  the  first  time  and  the  technology  is  constantly  changing.  Iterative  design  works 
best  if  it  is  part  of  the  overall  Web  management  process  rather  than  an  extra  chore 
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performed  every  year  or  so.  Between  iterations,  one  should  be  gathering  user  feedback, 
analyzing  log  data,  creating  rapid  prototypes,  and  testing  users. 

Part  of  the  iterative  design  should  also  be  an  evaluation  of  new  technologies  for 
the  next  version.  Such  a  technology  should  not  drive  the  change  but  rather  be  seen  as  a 
solution  to  known  usability  problems. 

4.  Collecting  Feedback  from  Field  Use 

One  of  the  easiest  and  most  effective  ways  to  improve  the  usability  of  a  Web  site 
is  to  see  what  real  users  do  with  it.  A  strength  of  the  Internet  is  the  ease  with  which  the 
users  can  provide  feedback.  A  common  practice  is  to  place  mail  to  and  feedback  buttons 
on  every  page,  encouraging  users  to  take  a  moment  to  send  the  developers  a  note  if  they 
have  a  problem,  a  complaint,  or  some  praise.  The  best  user  feedback  can  be  designed  into 
the  site  transparently. 

Good  logging  of  Web  site  usage  is  important  for  those  who  have  to  justify  the  cost 
of  developing  and  supporting  a  site.  As  far  as  improving  usability,  logs  can  also  indicate 
how  people  access  the  site,  what  problems  they  are  having,  and  how  a  redesign  affects 
usage. 


D.  HEURISTICS 

Heuristic  evaluation  is  one  of  the  most  important  aspects  of  usability  engineering: 
it  is  easy,  fast,  and  inexpensive.  It  involves  study  of  a  user  interface  by  a  small  set  of 
evaluators  for  violations  of  common  usability  principles  (rules  of  thumb,  heuristics)  so 
that  the  next  iterative  design  can  try  to  solve  the  usability  problems.  See  the  list  that 
follows  for  some  of  the  common  usability  heuristics  (Nielsen  1994). 
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•  Simple  and  natural  dialog. 

•  Speak  the  users’  language. 

•  Minimize  the  users’  memory  load. 

•  Consistency. 

•  Feedback. 

•  Clearly  marked  exits. 

•  Shortcuts. 

•  Good  error  messages. 

•  Prevent  errors. 

•  Help  and  documentation. 

Heuristic  evaluation  is  well  suited  to  the  Web,  where  everyone  is  in  a  huny  and 
the  site  can  be  evaluated  from  half-way  around  the  world.  When  doing  a  heuristic 
evaluation  for  the  Web,  the  common  heuristics  of  consistency,  feedback,  and  so  on  apply. 
But  a  few  other  rules  of  thumb  are  also  useful  when  evaluating  a  Web  user  interface. 

1.  User  Control 

Because  of  the  Web’s  architecture,  in  which  browsers  are  responsible  for 
interpreting  HyperText  Markup  Language  (HTML),  developers  never  really  know  who— 
or  what— will  be  processing  their  pages.  Forcing  users  into  particular  fonts,  sizes,  colors, 
screen  widths,  or  browser  versions  will  generally  reduce  usability.  Of  course,  designers 
need  to  have  some  control.  But  the  more  general  the  implementations  and  designs,  the 
more  likely  we  will  reach  satisfactory  levels  of  usability  for  the  widest  range  of  users  and 
over  the  longest  periods  of  time. 
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Emerging  Web  technologies  should  often  be  avoided  because  they  fail  on  this  user 
control  heuristic  early  in  their  life  spans.  One  example  is  animation.  To  accommodate  a 
diverse  set  of  situations,  users  should  be  allowed  to  start  and  stop  animation  on  their  own. 
On  one  hand,  animations  that  repeat  forever  are  distracting;  on  the  other,  those  that 
cannot  be  repeated  lose  their  value.  Until  users  gain  adequate  control  over  this  technology 
on  the  Web,  animations  should  be  used  very  carefully.  (Even  once  user  control  is  in 
place,  animation  should  be  used  only  when  appropriate  (Nielsen  1995). 

A  new  Web  feature  that  will  enable  greater  user  control  is  cascading  style  sheets 
(Andrews  1996).  Style  sheets  force  the  developer  to  separate  the  presentation  aspects  of 
the  page  from  the  content,  making  it  easier  for  a  user  to  "intercept"  the  presentation 
information  and  insert  his  or  her  own  requirements  for  display.  Beyond  selecting  link 
color,  for  example,  style  sheets  will  extend  the  amount  of  control  users  have  to  include 
many  other  items  of  personal  taste.  The  largest  gain,  however,  will  be  that  users  can 
specify  style  attributes  in  a  standard  way  to  ensure  their  own  personal  requirements  for 
usable  access.  For  example,  users  with  poor  eyesight  could  specify  large  font  sizes  for 
headers  and  text,  overriding  the  font  sizes  specified  by  the  authors. 

When  evaluating  a  site  for  user  control,  the  following  questions  can  be  asked  to 
determine  what  usability  problem  might  arise: 

•  Can  users  override  this  feature? 

•  Can  they  customize  to  suit  their  tastes  or  needs? 

•  Will  giving  users  control  of  this  feature  reduce  the  usability  of  the  site? 
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2.  Structure 


In  all  hypermedia  systems,  some  underlying  structure  is  important  to  help  users 
figure  out  where  they  are  and  where  they  can  go  next.  On  the  Web,  where  search  engines 
and  links  from  other  places  can  throw  users  into  the  middle  of  any  site,  showing  this 
structure  is  even  more  important.  Since  Web  browsers  lack  substantial  navigation  aids  to 
help  users  discover  this  structure,  Web  authors  have  to  do  a  lot  of  this  work  themselves. 
To  evaluate  structure,  one  can  look  at  every  major  page  of  a  site  and  ask  the  following 
questions: 


•  If  a  user  were  taken  directly  to  this  page  from  an  outside  site,  what  could  they 
figure  out  about  the  rest  of  the  site  from  this  one  page? 

•  Is  the  site  "brand"  present? 

•  Is  it  clear  which  part  of  the  site  they  are  in? 

•  Is  it  clear  how  to  navigate  to  other  parts  of  the  site? 

3.  Design  for  Change 

The  Web  is  about  change,  so  a  user  interface  had  better  be  able  to  deal  with  both 
continually  changing  content  and  design.  The  evaluator  can  ask  the  following  questions: 

•  How  is  old  content  archived? 

•  How  is  new  content  added? 

•  Can  this  design  withstand  the  addition  of  20  times  the  current  content? 
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If  everything  has  to  be  introduced  through  a  "What’s  new"  page,  the  site  was  not 
properly  designed  for  changing  content.  If  iterative  design  is  going  to  work  well,  the 
design  has  to  be  flexible  enough  to  withstand  small  changes  without  having  to  be  thrown 
away.  For  example,  if  user  feedback  indicates  a  need  for  an  additional  major  section  to  a 
Web  site,  can  the  design  handle  it,  or  would  the  site  have  to  be  redone  from  scratch?  If 
improving  the  usability  is  too  difficult,  it  simply  will  not  happen. 

E.  USABILITY  TRADEOFFS 

Usable  interface  design  involves  tradeoffs:  weighing  the  costs  and  benefits  to 
develop  the  best  solution  under  the  current  conditions.  The  Web  requires  some  very 
serious  tradeoff  decisions  because  of  its  low  bandwidth,  the  role  of  browsers,  and  the 
existence  of  standards. 

It  is  important  to  note  that  these  tradeoffs  are  not  meant  to  mandate  one  decision 
over  another.  What  is  important  is  to  understand  the  pros  and  cons  in  order  to  make  an 
informed  decision.  Sometimes  usability  has  to  be  sacrificed  because  of  cost,  time,  or 
management  issues,  but  blindly  adopting  a  new  Web  technology  rather  than  studying  the 
usability  tradeoffs  is  not  appropriate. 

For  example,  third  generation  site  design  (Lie  1996)  improves  usability  by 
providing  better  page  layout:  paragraphs  are  indented,  headers  are  close  to  their 
associated  paragraphs,  and  there  are  left  margins.  One  of  the  costs,  however,  is  that  access 
with  graphic  loading  turned  off  (to  speed  the  process)  presents  a  significantly  less  usable 
site;  the  precisely  aligned  text  is  no  more,  and  "unloaded  graphics"  icons  litter  the  screen. 
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So  if  one  is  willing  to  eliminate  use  by  one  segment  of  the  Internet  population,  a  third- 
generation  site  might  be  appropriate.  (Note  also  that  several  other  populations  of  users  are 
disenfranchised  by  third-generation  site  design  such  as  those  who  have  disabilities 
Paciello  1997).  Part  of  the  tradeoff  is  found  in  the  answers  to  the  following  questions:  is 
this  a  very  usable  site  for  a  few  users,  a  fairly  usable  site  for  the  majority  of  users,  or  a 
"good  enough"  site  for  all  users? 

1,  Bandwidth 

Access  speed  is  currently  one  of  the  major  constraints  on  design.  The  best  design, 
no  matter  how  good,  will  not  be  seen  if  it  takes  too  long  to  download.  Until  access  speed 
is  no  longer  a  major  problem,  tradeoffs  involving  the  quality  of  the  graphics,  the 
complexity  of  the  design,  and  the  use  of  higher  bandwidth  advanced  technologies  will  be 
important  to  understand  in  creating  usable  Web  sites.  In  addition,  developers  would 
benefit  from  observing  users  at  very  slow  speeds. 

Web  site  developers  must  be  cognizant  of  download  times  and  design 
accordingly.  According  to  a  newspaper  article  for  the  Seattle  Times  Web  site  (Williams, 
1998)  here  is  a  wish  list  for  Web  users  to  Web  designers: 

•  Create  pages  that  appear  on  the  screen  quickly.  Do  not  load  them  up  with  big 
graphics. 

•  Do  not  start  sound  files  running  without  asking  if  it  is  wanted  first. 

•  No  blinking,  animated  graphics,  or  dancing  objects. 

•  Make  navigation  easy.  Keep  the  buttons  in  the  same  place  as  you  move 
through  the  site.  Those  buttons  and  icons  should  be  logical  and  labeled. 


18 


Graphics  and  images  are  usually  the  largest  part  of  any  Web  page,  and  as  such  they  tend 
to  use  up  the  most  bandwidth.  Omitting  some  or  all  graphics  from  a  page  is  the  quickest 
and  easiest  way  to  decrease  download  time  of  a  Web  page,  although  some  smaller,  well 
chosen  graphics  may  make  the  page  more  readable.  Graphics  to  be  placed  on  a  Web  page 
should  be  chosen  carefully.  They  must  add  value  to  the  information  and  the  page  is  being 
used  to  convey,  not  just  be  placed  there  for  decorative  purposes.  Consider  that  if  a  page 
contained  ten  100KB  images,  the  download  amount  would  be  over  one  megabyte  of 
information.  This  certainly  does  not  bode  well  for  using  a  modem  connection. 

Another  tip  for  designers  to  consider  is  the  use  of  alternative  or  the  “AL=” 
attribute.  This  allows  the  user  to  have  a  description  of  an  object  while  it  is  loading  into 
the  browser.  If  the  transfer  is  interrupted  or  “times  out,”  the  user  will  know  if  the  item  is 
something  they  are  interested  in  trying  harder  to  obtain,  either  by  reloading  the  page  or 
returning  later  to  download  it  again.  This  also  describes  objects  to  those  who  have  image 
loading  turned  off  in  their  browsers,  or  those  who  have  browsers  that  do  not  support 
images.  Values  set  in  this  attribute  should  be  a  sensible  description  of  the  object  they 
represent.  There  are  four  types  of  images  contained  on  Web  pages;  page  toys,  navigation 
icons,  supplemental  or  interesting,  and  those  that  are  critical  for  understanding  the  page. 
(Flavell,  1998) 

2.  Standards 

Standards  make  the  World  Wide  Web  interoperable.  The  most  important  standard 
for  developers  is  HyperText  Markup  Language  (HTML).  The  latest  version  of  HTML  is 
4.0  (Raggett  and  Hertzbert  1996).  HTML  4.0  has  brought  the  major  Web  vendors 
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together,  making  the  Web  more  interoperable  and  slowing  the  "tag  wars"  that  were 
impeding  the  creation  of  usable  Web  sites  across  the  different  platforms.  Knowing  the 
HTML  standard  and  following  it  when  appropriate  are  crucial  in  making  informed  design 
and  implementation  decisions. 

In  general,  the  more  meticulous  we  are  in  following  the  standards,  the  more  usable 
a  site  is  for  the  largest  audience.  Any  variation  from  the  standard,  even  if  it  follows 
"Netscape’s  standard,"  will  have  an  effect  on  some  part  of  the  user  population.  It  would 
help  to  consider  the  following  questions: 

•  Is  that  new  tag  useful  enough  for  those  who  can  take  advantage  of  it? 

•  Does  it  significantly  affect  those  who  cannot  take  advantage  of  it? 

The  best  extensions  are  those  that  give  added  functionality  for  the  "haves"  while 
having  no  negative  effect  on  the  "have  nots."  A  good  example  of  this  is  the  background 
color  attribute  for  table  cells.  Because  they  are  not  part  of  the  older  HTML  3.2 
specification,  table  cell  background  colors  are  ignored  by  those  browsers  that  follow  the 
standard  and  have  not  implemented  this  extension.  Developers  need  to  be  aware  that  this 
is  indeed  not  part  of  the  standard  and  should  ensure  that  their  sites  are  still  usable  for 
those  who  do  not  have  access  to  this  added  feature. 

Developers  should  also  be  aware  of  cases  in  which  browsers  are  "forgiving"— i.e., 
those  that  create  usable  layouts  of  nonstandard  HTML.  For  example,  images  are  not 
allowed  in  PRE  sections  according  to  the  HTML  specification,  but  most  browsers  will 
generate  reasonable  interpretations  of  such  markup.  Designs  that  rely  on  such 
nonstandard  use  of  HTML  should  be  carefully  considered. 
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Designers  of  a  corporate  Intranet  have  it  much  easier  than  those  with  an  Internet 
audience.  If  all  users  are  known  to  have  a  particular  browser  and  version,  one  can  more 
safely  take  advantage  of  nonstandard  aspects.  Still,  any  nonstandard  usage  could  tie  that 
corporation  to  one  particular  vendor,  so  some  intelligent  decisions  need  to  be  made. 

3.  Browsers 

Another  tradeoff  occurs  because  of  the  crucial  role  that  browsers  play  on  the  Web. 
The  browser  does  certain  things  that  are  out  of  the  author’s  control-history,  bookmarks, 
and  exact  page  layout,  for  example.  The  fact  that  the  developer  has  to  rely  on  the  browser 
to  provide  certain  functionality  may  limit  the  creation  of  usable  sites.  One  need  only  look 
at  the  sophisticated  and  easy-to-use  interface  on  CD-ROMs  today  to  realize  the  limits  that 
browsers  impose  on  the  Web.  Browsers  often  have  bugs  that  make  certain  features 
unusable.  Netscape  Navigator’s  early  implementation  of  frames  had  serious  problems 
with  Back  and  bookmarking,  which  made  frames  quite  unusable.  (Now  those  bugs  are 
fixed  and  frames  are  more  usable— but  not  necessarily  usable  enough  for  certain 
applications.)  Since  not  everyone  has  the  most  current  version  of  browsers,  these  bugs  are 
still  an  issue  when  designing  a  site  today.  Finally,  there  are  a  lot  of  browsers,  each  acting 
slightly  different.  Developers  need  to  test  their  sites  on  as  many  platforms  as  possible  in 
order  to  ensure  usability  for  the  widest  range  of  visitors. 

F.  USABILITY  TESTING 

Solid  design  and  usability  are  features  explicitly  used  to  sell  products  to 
consumers.  When  we  hear  someone  selling  Windows  95™  as  being  "just  like  a  Mac,"  or 
an  Infiniti  salesman  preaching  that  thousands  of  customers  were  tested  in  order  to 
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determine  the  placement  of  displays  in  Infiniti  automobiles,  this  suggests  that  a  payoff 
exists  in  terms  of  customer  satisfaction  and  sales. 

There  are  methods  available  to  web  designers  that  may  not  have  been  applied 
previously  to  legacy  system  design.  Bellcore’s  User-Centered  Design  (UCD)  methodology 
helps  designers  to  rethink  and  redesign  existing  software  systems  to  accommodate  users 
and  the  web.  UCD  is  a  process  for  creating  software  that  meets  "documented  objectives 
for  usability  and  usefulness  (with)  an  early  and  continuous  focus  on  users  throughout  the 
analysis,  design,  and  development  process"  (Salasoo,  White,  Dayton,  Burkhart,  and  Root, 
1994).  It  is  never  too  late  to  start  usability  testing,  but  the  earlier  one  starts,  the  larger  the 
payoff  in  time  savings  and  user  satisfaction.  A  good  example  of  how  usability  testing  was 
applied  to  web  design  is  presented  by  Jakob  Nielsen  and  Darrell  Sano  (1995)  who  wrote 
about  their  interface  design  testing  of  the  new  Sun  Microsystems  web  site. 

A  user-friendly  web  interface  can  have  many  advantages  over  an  existing  legacy 
interface.  These  advantages  include: 

•  Reduce  errors  due  to  a  redesigned. 

•  Increased  user  acceptance  of  the  system. 

•  Increase  flexibility  for  changes  and  improvement  to  the  system. 

•  Increase  number  of  platforms  supported  with  a  consistent  interface  across 
platforms. 

G.  CONCLUSION 

Usability  engineering  techniques,  such  as  user  testing,  iterative  design,  heuristic 
evaluation,  and  user  feedback,  can  make  Web  sites  easier  to  use.  Usability  evaluations  of 
new  and  emerging  Web  technologies,  such  as  frames,  JavaScript,  Java  and  animated  GIFs 
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(Graphics  Interchange  Format),  are  particularly  important  so  that  the  tradeoffs  of 
incorporating  them  into  a  site  can  be  understood.  Web  standards,  such  as  HTML  4.0, 
provide  the  basis  of  interoperability  and  help  to  ensure  that  designs  will  remain  usable 
over  time.  Although  excitement  about  the  Web  is  still  motivating  users  to  play  with  new 
technologies,  along  with  the  increased  use  of  the  Web  for  "everyday"  tasks  and  not  just 
for  fun,  indicates  a  trend  in  which  "what’s  useful"  will  become  more  important  than 
"what’s  cool."  Those  sites  that  incorporate  usability  engineering  into  their  development 
process  will  find  that  they  are  better  able  to  survive  in  the  highly  competitive  marketplace 
of  the  World  Wide  Web. 
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III.  METHODOLOGY 


A.  SUBJECTS 

1.  Background 

Selection  of  participants  in  this  research  focused  on  potential  users  of  the  Aviation 
Command  Safety  Assessment  (ACSA)  Web  site  and  consisted  of  aviation  units  from 
Norfolk,  Virginia  and  San  Diego,  California.  This  focus  limited  the  research  to 
Helicopter  Combat  Support  Services  squadrons  HC-2,  HC-3  and  HC-11,  Light  Airborne 
Multi-Purpose  System  (LAMPS)  squadron  HSL-45  and  Airborne  Early  Warning  (AEW) 
squadrons  VAW-120  and  VAW123.  These  squadrons  comprised  a  mix  of  rotary  and 
fixed  wing  pilots. 

2.  Participants 

From  the  available  pilots  in  the  squadrons,  several  were  asked  to  participate  in 
either  the  General  Web  Survey  or  usability  testing  of  Aviation  Command  Safety 
Assessment  (ACSA)  Survey.  All  test  subjects  elected  to  participate  voluntarily.  No 
individuals  with  specific  experience  in  the  use  of  computers,  software  or  the  web  site 
design  were  singled  out  for  the  purpose  of  including  or  excluding  them  from  the 
evaluation.  All  participants  received  an  overview  of  the  Aviation  Command  Safety 
Assessment  Web  site  goals  and  objectives  and  an  explanation  of  their  role  in  the  usability 
testing.  The  participants  were  asked  to  be  candid  in  their  response  to  the  General  Web 
site.  Aviation  Command  Safety  Assessment  (ACSA)  Usability  Survey,  and  interview  as 
their  responses  would  be  confidential. 
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B.  INSTRUMENTATION 


1.  Questionnaire 

A  General  Web  Survey  was  given  to  squadron  volunteers  in  an  attempt  to  assess 
the  amount  of  integration  of  computer  into  the  lives  of  aviation  personnel.  The  intent  of 
this  survey  was  to  profile  the  aviation  user.  By  determining  the  level  of  familiarity  of  the 
target  group  with  current  computer  technology,  a  baseline  can  be  established  by 
determining  the  least  common  denominator.  Questions  in  this  survey  were  designed  to  be 
as  neutral  as  possible  to  determine  the  true  user  is  perspective.  The  five  basic 
characteristics  of  questions  and  answers  that  are  fundamental  to  a  good  measurement 
process  (Fowler,  1995)  are  as  follows: 

•  Questions  need  to  be  consistently  understood. 

•  Questions  need  to  be  consistently  administered  or  communicated  to 
respondents. 

•  What  constitutes  an  adequate  answer  should  be  consistently  communicated. 

•  Unless  measuring  knowledge  is  the  goal  of  the  question,  all  respondents 
should  have  access  to  the  information  needed  to  answer  the  question 
accurately. 

•  Respondents  must  be  willing  to  provide  the  answer  called  for  in  the  question. 


How  to  design  questions  to  measure  subjective  states  (Fowler,  1995): 

Because  there  are  no  standards  against  which  to  evaluate  the 
correctness  or  rightness  of  answers,  standardization  of  the 
stimulus  of  the  question  is  particularly  critical  in  measuring 
subjective  states.  For  this  reason,  designing  questions  that  can 
be  administered  in  a  consistent  way  and  that  mean  the  same 
thing  to  all  respondents,  to  the  extent  possible,  is  high  on  the  list 
of  strategies  for  creating  good  measurement  of  subjective  states. 
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Equally  important  is  standardizing  the  response  task.  That 
means  clearly  defining  the  dimension  or  continuum  respondents 
are  to  use  in  their  rating  task  and  giving  them  a  reasonable  way 
to  replace  themselves,  or  whatever  else  they  are  rating,  on  that 
continuum. 

Fowler  continues,  the  answers  that  come  from  a  subjective  questions  have  no 
absolute  meaning,  rather  they  are  relative.  The  position  of  the  answers  relative  to  each 
other  is  where  the  relevant  information  is  found.  In  general,  surveys  should  ask  things 
that  respondents  are  able  to  reliably  report. 

The  first  survey  included  general  demographic  questions  about  user  characteristics 
to  determine  what  the  subjects  had  access  too,  and  what  they  were  accustom  to  using. 
There  were  also  some  questions  based  on  a  five  point  Lakers  scale  (Fowler  1995) 
regarding  Web  sites  on  the  Internet  itself,  as  well  as  questions  relating  to  the  time  and 
amount  of  time  spent  on  Web  access.  This  survey  was  handed  out  to  the  participants  and 
collected  the  next  day. 

The  Aviation  Command  Safety  Assessment  (ACSA)  Usability  Survey  was 
designed  to  ascertain  input  about  the  ACSA  Web  site  specifically.  Questions  were  asked 
after  the  participants  had  on  opportunity  to  explore  the  site.  The  first  section  of  this 
particular  survey  was  a  set  of  general  questions  about  the  site  based  on  a  five  point  Likert 
scale.  The  second  part  was  interactive,  in  which  participants  accessed  a  Web  page  and 
were  asked  questions  about  the  pages,  they  directly  observed.  This  portion  was  executed 
on  a  one-on-one  basis,  with  the  pages  being  accessed  for  the  individual  in  an  attempt  to 
standardize  the  exercise.  Participants  were  not  asked  to  go  to  the  site  and  find  out 
answers  to  specific  questions  as  in  the  study  by  Morkes  and  Nielsen  (1997)  on  usability. 
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This  study  concluded  that  by  writing  in  concise,  objective  and  scannable  format, 
designers  can  greatly  increase  usability  of  a  Web  page. 

In  order  to  assure  anonymity,  no  names  were  taken  and  the  data  shall  be  displayed 
on  in  aggregate  form.  Appendices  A  and  B  show  the  General  Web  survey  and  the  ACSA 
Usability  Survey,  respectively,  while  raw  summary  data  are  shown  for  each  respective 
survey  in  Chapter  IV. 

2.  Interviews 

All  participants  were  interviewed  upon  completion  of  the  Aviation  Command 
Safety  Assessment  (ACSA)  Usability  Survey.  As  with  the  comments  section  of  the 
questionnaire,  the  interviews  were  designed  primarily  to  probe  for  frustration  with 
specific  parts  of  the  ACSA.  Only  four  questions  were  asked: 

1.  Did  you  find  the  portion  of  the  ACSA  that  you  used  to  be  an  easy  tool  for 
completing  the  survey? 

2.  Was  there  anything  that  you  found  particularly  difficult  to  do  or 
understand? 

3.  If  you  could  change  one  thing  about  the  ACSA  Web  site  to  make  it  easier 
to  use,  what  would  it  be? 

4.  Are  there  any  features  you  would  like  to  see  added  to  the  ACSA  Web  site? 
Significant  responses  to  these  questions  are  detailed  in  Appendix  D. 

C.  PROCEDURES 

A  pretest  walkthrough  of  log-on  procedures  was  completed  prior  to  the  field 
evaluation.  The  walkthrough  took  place  in  several  computer  systems  located  at  the  Naval 
Aviation  Safety  School  and  reflected  the  usability  test  as  planned  for  the  actual  testing  of 
the  ACSA  Web  site.  Participants  for  this  phase  were  staff  volunteers  representing  typical 
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end  users  in  the  fleet.  During  the  pretest  walkthrough,  questions  to  log-on  procedures  and 
interview  questions  were  refined. 

The  first  usability  test  of  the  Aviation  Command  Safety  Assessment  was  done  at 
HC-1 1  San  Diego,  California.  The  volunteer  participants  were  given  a  brief  presentation 
on  the  ACSA  Web  site  and  an  explanation  of  what  was  desired  from  them  during  the  test 
(Appendix  C).  It  was  expressly  noted  to  each  participant  that  they  were  not  the  subject  of 
the  research  but  were  only  assisting  in  the  evaluation  of  the  ACSA  Web  site.  After  the 
presentation  the  subjects  were  scheduled  to  use  the  ACSA  Web  site  so  not  to  interfere 
with  their  other  daily  tasks. 

Prior  to  beginning  each  individual  session  the  participants  were  familiarized  with 
the  log-on  procedures  (Appendix  B).  The  previous  instructions  were  reiterated  and  the 
desire  that  the  participants  talk  aloud  during  the  session  and  point  to  the  screen  to  indicate 
problems  was  stated.  Participants  were  then  instructed  to  proceed  reviewing  the  Web  site 
using  the  ACSA  Usability  Survey  as  a  guide.  Total  time  to  complete  the  test  was  noted 
so  that  a  basis  for  future  tests  could  be  established. 

D.  ANALYSIS  STRATEGY 

Descriptive  statistics  were  used  to  examine  the  data  from  the  questionnaire.  The 
number  and  percentage  of  subjects  who  answered  a  question  in  a  specific  way  is  shown  in 
a  frequency  table  and  illustrated  with  a  bar  graph.  The  number  of  participants  in  the  test 
were  used  as  a  base  for  calculating  the  percentages.  The  mean,  median  and  standard 
deviation  were  also  calculated  for  the  answers  to  each  question.  All  tabulation  and 
calculations  were  accomplished  using  Microsoft  Excel  spreadsheet  software. 
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Significant  comments,  taken  from  the  open  ended  portion  of  the  questionnaires 


and  interviews,  are  quoted  in  Appendices  D  and  E  respectively. 
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rv.  RESULTS  AND  DISCUSSION 


A.  DATA  ANALYSIS 

The  author  conducted  a  survey  of  the  aviation  participants  as  described  in  the 
“survey  Methodology”  section  of  Chapter  HI  of  this  thesis.  The  survey  was  conducted  at 
each  aviation  squadron’s  command  as  noted  early.  The  survey  was  deemed  to  be  self- 
explanatory,  but  the  author  was  available  to  answer  any  questions.  An  early  question 
concerned  the  meaning  of  intuitive.  As  found  in  Webster’s  New  World  Dictionary, 
intuitive  is  defined  as  of,  pertaining  to,  or  arising  from  intuition-,  with  intuition  being  the 
act  or  faculty  of  knowing  without  the  use  of  use  of  rational  processes;  immediate 
cognition.  Participants  returned  fourteen  of  the  fourteen  surveys  given  out. 

The  findings  presented  in  this  section  have  been  taken  from  the  General  Web 
Survey  and  Aviation  Command  Safety  Assessment  (ACSA)  Usability  Survey  and 
interviews  with  the  participants. 

The  responses  to  the  questionnaires  were  transferred  to  a  Microsoft  Excel 
spreadsheet  (Appendices  E  and  F  respectively)  and  simple  descriptive  statistics  were 
computed  from  the  questions  that  used  Likert  scales.  The  resulting  data  were  converted 
to  tables  and  figures.  Responses  to  the  non-scaled  questions  were  reviewed  for  any 
significant  differences  or  similarities  among  the  respondents. 

Like  the  responses  to  the  non-scaled  questions  of  the  questionnaire,  responses  to 
interviews  questions  (Appendix  D)  were  examined  for  trends  indicating  problems  or 
frustrations  experienced  across  participants. 
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B.  FINDINGS 


1.  General  Web  Survey 

The  following  pages  present  a  detail  statistical  and  graphical  breakdown  of 
responses  to  each  question  of  the  questionnaire.  The  first  five  questions  were  used  to 
collected  a  profile  of  the  squadron  user,  while  questions  six  through  twenty-one  focused 
on  computer  usage. 

Sixty-four  percent  of  the  aviators  tested  in  this  survey  indicated  they  spend 
between  eight  to  ten  hours  at  work  per  day.  In  addition  the  same  percent  indicated  they 
are  between  twenty-five  and  twenty-seven  years  of  age.  Gender  was  not  equally 
represented  with  males  making  up  seventy-eight  of  the  participants,  however  this  is  a 
good  representation  of  a  typical  squadron  with  respect  to  gender.  All  participants 
indicated  they  are  in  the  United  States  Navy  with  forty-two  percent  of  them  indicating  a 
rank  of  Lieutenant  (03)  twenty-one  percent  as  Lieutenant  Junior  Grade  (02)  and  twenty- 
one  percent  indicating  a  rank  of  Lieutenant  Commander  (04).  See  Figures  4. 1  through 
4.5  for  a  detailed  statistical  and  graphical  breakdown  of  responses  to  each  question  of  the 
questionnaire. 

Question  1.1:  How  many  hours  per  day,  on  average  do  you  spend  at  work? 


Figure  4.1.  Hours  worked 
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Question  1.2:  What  is  your  age? 


Figure  4.2.  Age 


Question  1.3:  What  is  your  sex? 


Figure  4.3.  Sex 

Question  1 .4:  What  is  your  branch  of  service? 


Figure  4.4.  Branch  of  Service 


Number  ot  respona 


Question  1.5:  What  is  your  pay  grade? 


Figure  4.5.  Pay  grade 


All  of  the  respondents  indicated  using  a  computer  at  home,  with  fifty  percent 
doing  so  at  work  and  thirty  percent  using  a  mobile  computer.  All  reported  using  personal 
computers  (PCs),  twenty-one  percent  use  a  Macintosh,  and  twenty-eight  use  a  notebook 
computer.  Seventy  percent  use  Windows  95/98™,  with  fifty  percent  indicating  then  use 
Microsoft  Office  95/97.  Browsers  use  was  split  evenly  with  fifty  percent  using  either 
Microsoft  Internet  Explorer  or  Netscape.  Obviously  some  of  the  respondents  use  two 
different  browsers,  but  obtaining  preference  was  not  the  goal  of  this  question.  See 
Figures  4.6  and  4.7  for  a  graphical  breakdown  of  responses  to  each  question  of  the 
questionnaire. 


Question  1.6: 1  use  a  computer  at?  Question  1.7: 1  use  computer  for  what? 


Figure  4.6.  Computer  use  Figure  4.7.  Computer  Application 
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Activities  perform  on  computer  include:  E-mail  and  Web  access  or  Web  browsing 
being  performed  by  all  respondents;  word  processing  being  used  by  seventy-one  percent 
of  the  participants.  Other  applications  include,  forty-two  percent  using  presentation 
graphics  and  twenty-one  percent  using  financial  programs  and  twenty-one  percent 
indicating  they  play  games.  A  small  percent  indicated  that  they  used  the  computer  to 
perform  spreadsheet  or  DoD  applications. 


Question  1.8:  What  type  of  activities  do  you  perform  on  computers? 


Figure  4.8.  Computer  Activities 

Sixty-four  percent  of  the  respondents  indicated  that  they  had  never  completed  a 
formal  computer  course.  Over  half  of  the  respondents,  fifty-seven  percent  rated  their 
computer  skills  as  intermediate,  while  twenty-eight  percent  indicated  themselves  as 
novice.  Question  1.9:  Have  you  ever  completed  a  formal  computer  course? 


Yes  No 


Figure  4.9.  Formal  course 
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Question  1.10:  How  would  you  rate  your  computer  skills? 


Novice  intermediate  Expert 


Figure  4.10.  Computer  skills 

Time  spend  on  the  Internet  was  distributed  between  zero  to  thirty,  thirty  to 
sixty  minutes  and  one  to  three  hours,  each  of  these  categories  received  fifty  percent, 
thirty-five  percent  and  fifteen  percent  respectively.  Sixty-four  percent  indicated  they 
browse  or  search  the  Web  several  times  daily  and  thirty-six  percent  classified  themselves 
at  once  daily.  Seventy-one  percent  spend  two  to  ten  minutes  on  a  Web  site  once  they 
reach  it,  twenty-one  percent  spend  zero  to  two  minutes,  and  eight  percent  spend  ten  to 
thirty  minutes  at  a  site  once  they  get  there. 

Question  1.1 1 :  In  total,  approximately  how  much  time  to  you  spend  on  the  internet? 


0-30  m  in  30-60  m  in  1-3  hrs  >  3  hrs 


Figure  4.11.  Internet  usage 
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Question  1.12:  Approximately  how  often  do  you  Browse  or  Search  the  Web? 
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Several  time  daily  Once  daily  <5times  per  week 


Figure  4.12.  Web  access  time 


Question  1.13:  When  you  get  on  a  Web  site,  how  long  do  you  stay  there? 


Figure  4.13.  Web  site  visit  time 


Table  4.1  shows  the  ranking  for  actions  taken  once  the  respondents  are  finished 
looking  at  a  Web  site.  Seven  out  of  fourteen  respondents  indicated  that  then  jump  to 
another  Web  site  without  using  a  link  as  the  most  preferred  action.  The  second  most 
preferred  action  indicated  by  the  respondents  is  to  follow  a  link  to  another  Web  site. 
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Question  1.14.  When  you  are  finished  looking  at  a  web  site,  do  you  (rank  in 
order,  with  1  being  most  used  action): 
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Table  4.1  Most  Popular  Action  Taken  When  Finished  Viewing  a  Web  Site 


Each  user  was  asked  to  rank  order  common  web  page  problems  that  detract  from 


the  page’s  usability  the  most.  Table  4.2  shows  the  overall  results  to  this  question. 


Question  1.15.  Regarding  the  graphic  user  interface  used  on  Web  sites  you 
visit,  what  do  you  find  most  detracting?  (rank  in  order,  with  1  being  most  used 
action): 
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Table  4.2  Regarding  GUI  used  on  Web  Site  -  Most  Detracting 


This  data  confirms  Nielsen’s  points  about  Web  site  design.  It  also  verifies  that  aviation 


personnel  want  many  of  the  same  features  as  the  users  in  Nielsen’s  usability  test. 
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Complexity  of  a  Web  site’s  address  was  not  viewed  as  a  discouraging  factor  in 
whether  or  not  participants  visited  or  used  that  (Figure  4.14  and  Table  4.3). 


Question  1.16:  The  complexity  of  a  Web  site’s  address  discourages  me  from 
visiting/using  that  site.? 


Table  4.3  Response  Statistics  for  Question  1.16 
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Mean 

2.36 
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0.928 

The  next  three  questions  explored  how  participants  view  computers  with  respect 
to  the  workplace.  Twelve  of  the  fourteen  respondents  indicated  that  computers  do  not 
make  them  more  productive  (Figure  4.15  and  Table  4.4).  Half  of  the  participants 
indicated  that  computer  increase  their  workload  (Figure  4.16  and  Table  4.5).  Eight  of  the 
fourteen  participants  disagree  strongly  that  computer  reduce  paperwork  (Figure  4.17  and 
Table  4.6). 
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Question  1.17:  Computers  make  me  more  productive? 
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Figure  4.15.  Productivity 


Table  4.4  Response  Statistics  for  Question  1.17 


<V  V\ v  ^  J 

Value 

Mean 

1.86 

Median 

2.00 

Std  Deviation 

0.662 

Question  1.18:  Computer  increase  my  workload? 


Figure  4.16.  Workload 

Table  4.5  Response  Statistics  for  Question  1.18 
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Question  1.19:  Computers  reduce  paperwork? 
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Figure  4.17.  Computers  Reduce  Paperwork 


Table  4.6  Response  Statistics  for  Question  1.19 


.'■VihBE'8'. 

Mean 

1.79 

Median 

1.00 

Std  Deviation 

1.05 

General  questions  were  asked  in  which  the  participants  could  write  in  responses 
(Appendix  E).  The  first  of  these  was  to  list  three  Web  sites  they  like  and  give  the  reasons 
why.  Eleven  of  the  fourteen  responses  involved  news  related  sites  like  ABC,  CNBC, 
CNN,  the  Weather  Channel,  the  Washington  Post  and  ZDNet,  while  six  participants 
specifically  listed  ESPN.  The  reoccurring  comment  for  ESPN  site  was  that  it  contained 
good  links  to  many  other  sites  of  interest.  There  were  two  references  to  search  engines, 
due  to  the  weath  of  informaiton  possible  there.  Other  sites  that  were  not  preferred  were 
shunned  because  of  broken  links,  promise  but  no  substance,  gratuitious  graphics,  most 
news  sites  for  not  being  interactive,  and  search  engines  for  being  low  yield.  Other  sites 
were  singled  out  for  requiring  too  much  time  to  find  the  desired  information  (i.e.  non- 
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intuitive  sites),  non-current  information,  too  much  advertising,  and  sites  that  were  not 
connected  or  hyperlinked. 

2.  ACSA  Specific  Survey 

In  the  first  section  of  this  survey,  participants  were  asked  to  respond  to  a  set  of 
statements  specifically  regarding  the  ACSA  Web  site.  The  first  series  of  questions  was  to 
get  an  overall  impression  about  the  ACSA  Web  site.  The  following  pages  present  a 
detailed  statistical  breakdown  of  responses  to  each  question  of  the  questionnaire.  The 
first  seven  questions  focused  on  information  content  of  the  ACSA  Web  site.  These  seven 
questions  were  used  to  explore  participants’  perception  of  whether  the  ACSA  Web  site 
would  the  needs  of  the  user.  Most  participants  indicated  that  the  ACSA  Web  site 
provided  information  that  was  relevant,  accurate,  complete,  current,  required,  intuitive 
and  easy  to  read.  Figure  4. 18  through  Figure  4.24  and  Table  4.8  through  Table  4. 14  show 
the  statistical  breakdown. 


Question  2.1:  Information  on  the  site  is  relevant? 


Figure  4.18.  Relevant  Information 
Table  4.7  Response  Statistics  for  Question  2.1 
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Question  2.2:  Information  on  the  site  is  accurate? 


2  Disagree  Neither  Agree 

Strongly  Disagree  Strongly 

nor  Agree 


Figure  4.19.  Accurate  Information 


Table  4.8  Response  Statistics  for  Question  2.2 


Question  2.3:  Information  on  the  site  is  complete? 


Figure  4.20.  Complete  Information 
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Table  4.9  Response  Statistics  for  Question  2.3 


Question  2.4:  Information  on  the  site  is  current? 


Disagree  Neither  Agree 

Strongly  Disagree  Strongly 

nor  Agree 


Figure  4.21.  Current  Information 

Table  4.10  Response  Statistics  for  Question  2.4 


Mean 

3.94 

Median 

4.00 

Std  Deviation 

0.639 

Question  2.5:  Information  on  the  site  meets  my  needs? 


Disagree  Neither  Agree 

Strongly  Disagree  Strongly 

nor  Agree 


Figure  4.22.  Information  Meets  Needs 


Table  4.11.  Response  Statistics  for  Question  2.5 


;  Statistic  -  '  — 

Mean 

3.67 

Median 

4.00 

Std  Deviation 
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Question  2.6:  Information  on  the  site  is  intuitive? 


Disagree  Neither  Agree 

Strongly  Disagree  Strongly 

nor  Agree 


Figure  4.23.  Information  Intuitive 
Table  4.12.  Response  Statistics  for  Question  2.6 


Mean  3.22 

Median  3.00 

Std  Deviation  0.732 


Question  2.7:  Information  on  the  site  is  easy  to  read? 


Figure  4.24.  Information  Easy  to  Read 
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Questions  concerning  general  appearance  of  the  ACSA  Web  site  indicated  general 
satisfaction.  Site  address  was  not  considered  too  complex  (Figure  2.25  and  Table  4.15). 
Most  agreed  that  scrolling  was  not  too  excessive  (Figure  2.26  and  Table  4.16).  Link 
colors  and  colors  used  throughout  the  Web  site  were  acceptable  (Figure  2.27  and  Figure 
2.28,  Table  4.17  and  Table  4.18).  The  amount  of  graphics  used  received  mixed  opinions 
but  generally  the  Web  site  appears  to  have  a  good  blend  (Figure  2.29  and  Table  2.19), 
however  the  user  indicated  that  they  generally  did  not  want  additional  graphics  added 
(Figure  2.30  and  Table  2.20). 

Question  2.8:  The  site  address  is  too  complex? 


Disagree  Neither  Agree 

Strongly  Disagree  Strongly 

nor  Agree 


Figure  4.25.  Address  Complexity 

Table  4.14  Response  Statistics  for  Question  2.8 


Question  2.9:  The  amount  of  page  scrolling  is  acceptable? 


Disagree  Neither  Agree 

Strongly  Disagree  Strongly 


nor  Agree 


Figure  4.26.  Page  Scrolling 


Table  4.15  Response  Statistics  for  Question  2.9 


Question  2. 10:  The  link  colors  are  acceptable? 


Disagree  Neither  Agree 

Strongly  Disagree  Strongly 

nor  Agree 


Figure  4.27.  Link  Colors 

Table  4.16  Response  Statistics  for  Question  2.10 


Mean 
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4.00 

Std  Deviation 
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Question  2.11:  The  colors  used  throughout  the  site  are  acceptable? 


Disagree  Neither  Agree 

Strongly  Disagree  Strongly 

nor  Agree 


Figure  4.28.  Site  Colors 

Table  4.17  Response  Statistics  for  Question  2. 1 1 
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Question  2.12:  The  amount  of  graphics  is  acceptable? 
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Figure  4.29.  Graphics 


Table  4.18  Response  Statistics  for  Question  2.12 
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Question  2.13: 1  would  like  to  see  more  graphics. 
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Figure  4.30.  More  Graphics 


Table  4.19  Response  Statistics  for  Question  2.13 
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Mean 

2.78 

Median 

3.00 

Std  Deviation 
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The  next  two  question  concerned  down  load  times  between  HTML  and  Java 
applets.  Download  times  for  the  HTML  format  are  generally  acceptable  (Figure  4.31  and 
Table  4.20),  however,  Java  applet  download  times  were  viewed  as  unacceptable  (Figure 
4.32  and  Table  4.21). 

Question  2.14:  Download  times  are  acceptable  (HTML  format). 
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Figure  4.31.  HTML  Download  Times 
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Table  4.20  Response  Statistics  for  Question  2.14 


Mean 

4.50 

Median 

4.50 

Std  Deviation 

0.514 

Question  2.15:  Download  times  are  acceptable  (Survey  Applet). 
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Figure  4.32.  Survey  Applet  Download  Times 


Table  4.21  Response  Statistics  for  Question  2.15 
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Question  concerning  navigation  through  the  ACSA  Web  site  received  less 
favorable  response.  Navigation  support  was  considered  to  be  lacking  in  that  there  were 
no  links  to  questions  or  common  problems  (Figure  4.33  and  Table  4.22).  Navigation 
from  one  web  page  to  another  was  considered  clear  (Figure  4.34  and  Table  4.23). 
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Question  2.16:  Navigation  support  is  acceptable. 


nor  Agree 


Figure  4.33.  Navigation  Support 
Table  4.22  Response  Statistics  for  Question  2.16 


Question  2.17:  Navigation  through  the  site  is  intuitive. 
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Figure  4.34.  Navigation  Intuitive 
Table  4.23  Response  Statistics  for  Question  2. 17 
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4.00 

Std  Deviation 

0.485 

The  next  series  of  questions  centered  around  individual  Web  page  design  and  overall 
presentation.  These  questions  focused  attention  on  the  seven  pages  which  comprise  the 
ACSA  Web  site.  Web  site  design  and  overall  presentation  of  for  each  page  was  rated  and 
generally  received  favorable  marks.  The  Main  page  design  and  overall  presentation 
received  positive  marks  (Figure  4.35  and  Table  4.24).  The  HTML  Survey  page 
receieved  a  very  good  rating  with  ten  of  the  eighteen  respondents  rating  it  at  veiy  good  ( 
Figure  4.36  and  Table  4.25). 


Question  2.18:  Main  page  design/overall  presentation. 


Figure  4.35,  Main  Page  Design 


Table  4.24  Response  Statistics  for  Question  2.18 
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Question  2.19:  HTML  survey  page  design/overall  presentation. 


Figure  4.36.  HTML  Survey  Page  Design 


Table  4.25  Response  Statistics  for  Question  2.19 
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Mean 
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Std  Deviation 
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The  Survey  Applet  page  design  received  mixed  rating  with  five  of  the  eighteen 
participants  indicating  it  was  a  poor  design.  Primary  reason  for  such  a  low  rating  was 
primary  due  to  flashing  text,  undulating  text,  and  scrolling  pages  (Figure  4.37  and  Table 
4.26).  Several  participants  indicated  that  downloading  the  Survey  Applet  was  excess  and 

often  timed  out  due  to  server  none  response  errors. 

Question  2.20:  Survey  Applet  page  design/overall  presentation. 


Poor  Acceptable  Good  Very  Good  Excellent 


Figure  4.37.  Survey  Applet  Page  Design 
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Table  4.26  Response  Statistics  for  Question  2.20 
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Participants ’s  responses  to  design  of  the  Results  Applet  page  indicated  gereral 
satisfaction,  however,  eight  participants  left  this  question  blank  due  to  non-access  to  the 
page.  Several  questions  did  arise  during  the  interview  which  question  why  squadron 
personnel  would  have  access  to  the  survey  data.  This  page  was  designed  to  give 
Commanding  Officer  the  ability  to  view  results  of  a  question  and  make  comparisons  with 
other  units.  Due  to  the  limited  availablity  of  Commanding  Officers  to  participant  in  the 
ACSA  usability  survey  it  was  included  soley  in  a  effort  to  evaluate  it  as  a  useful  tool  in 
the  future.  Ten  of  the  eighteen  responses  are  presented  in  Figure  4.38  and  Table  4.27. 

Question  2.21:  Results  Applet  page  design/overall  presentation. 


Poor  Acceptable  Good  Very  Good  Excellent 


Figure  4.38.  Results  Applet  Page  Design 
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Table  4.27  Response  Statistics  for  Question  2.21 


Mean 
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3.00 

Std  Deviation 

0.567 

Participants’  responses  to  questions  concerning  the  design/overall  presentation  of 

the  Help  page  was  acceptable  (Figure  4.39  and  Table  4.28).  Design  and  overall 

presentation  with  respect  to  Secure  HTML  Survey  page  received  mixed  results  due  to  a 

lack  of  understanding  between  the  HTML  Survey  page  and  Secure  HTML  Survey  page 

(Figure  4.40  and  Table  4.29).  Participants’  response  to  design  and  overall  presentation  of 

the  ACSA  newsgroup  page  indicated  a  lack  of  understanding  of  what  the  page  was  design 

for  (Figure  4.41  and  Table  4.30).  Several  responses  to  these  two  questions  were  left 

blank  and  only  eleven  participants  indicated  a  response. 

Question  2.22:  Help  page  design/overall  presentation. 


Poor  Acceptable  Good  Very  Good  Excellent 


Figure  4.39.  Help  Page  Design 


Table  4.28  Response  Statistics  for  Question  2.22 
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Std  Deviation 

0.583 
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Question  2.23:  Secure  HTML  Survey  design/overall  presentation. 


Figure  4.40.  Secure  HTML  Survey  Page  Design 


Table  4.29  Response  Statistics  for  Question  2.23 


Question  2.24:  ACSA  newsgroup  design/overall  presentation. 


Figure  4.41.  ACSA  Newsgroup  Page  Design 


Table  4.30  Response  Statistics  for  Question  2.24 
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The  next  series  of  questions  were  to  have  each  participant  rate  each  page 

according  to  functionality  and  ease  of  use.  Instructions  of  Main  page  received  only  minor 

favorable  results  due  too  the  technical  tone  of  instructions  and  lack  of  any  statements 

concern  confidentiality  (Figure  4.42  and  Table  4.31). 

Question  2.25:  Functionality  &  Ease  of  use  for  Instructions  on  Main  page. 
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Figure  4.42.  Main  Page  Functionality 


Table  4.31  Response  Statistics  for  Question  2.25 
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The  next  two  questions  compared  two  different  survey  formats,  HTML  and  Java 
Applet.  Participants  indicated  a  preference  for  the  HTML  survey  for  several  reasons, 
such  as,  easy  of  reading,  download  time,  color  separation  for  each  question  and  not 
having  too  scroll  to  read  each  question  (Figure  4.43  and  Table  4.32). 
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Question  2.26:  Functionality  &  Ease  of  use  for  HTML  Survey  page. 


Poor  Acceptable  Good  Very  Good  Excellent 


Figure  4.43.  HTML  Survey  Functionality 
Table  4.32  Response  Statistics  for  Question  2.26 


Mean 

4.17 
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Std  Deviation 

0.618 

A  majority  of  the  participants  indicated  that  the  Survey  Applet  format  was 
acceptable  but  not  as  good  as  the  HTML  format.  Responses  during  the  interview 
indicated  that  due  to  fashing  text,  alignment  while  reading  and  selecting  the  answer, 
layout  of  page  and  scrolling  to  read  each  question  contributed  to  negative  views  (Figure 
4.44  and  Table  4.33). 


Question  2.27:  Functionality  &  Ease  of  use  for  Survey  Applet  page. 


Figure  4.44.  Survey  Applet  Functionality 
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Table  4.33  Response  Statistics  for  Question  2.27 
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Participants’s  responses  to  functionality  and  ease  of  use  of  the  Results  Applet 
page  indicated  gereral  satisfaction,  however,  eight  participants  left  this  question  blank  due 
to  non-access  to  the  page.  Many  participants  indicated  during  the  interview  that  the 
Results  Applet  was  an  impressive  tool  for  Commanding  Offiers  to  use.  Several 
participants  made  postive  comments  on  the  ability  to  view  data  in  many  different  queries. 
Ten  of  the  eighteen  responses  are  presented  in  Figure  4.45  and  Table  4.34. 

Question  2.28:  Functionality  &  Ease  of  use  for  Results  Applet  page. 


Figure  4.45.  Results  Applet  Functionality 


Table  4.34  Response  Statistics  for  Question  2.28 


Mean 

3.70 

Median 

4.00 

Std  Deviation 

0.483 
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The  Help  page  was  not  hard  to  understand  and  received  favorable  marks, 
however,  participants  indicated  a  need  for  a  table  of  contents  and  links  to  major  points  in 
the  help  page.  Several  participants  challenged  the  naming  of  Help  page  while  providing 


nothing  more  than  a  frequently  asked  questions  page  (Figure  4.46  and  Table  4.35). 
Question  2.29  Functionality  &  Ease  of  use  for  Help  page. 


Figure  4.46.  Help  Functionality 
Table  4.35  Response  Statistics  for  Question  2.29 


■v.  •'  ;;g  Value  ^ 

Mean 

3.44 

Median 

3.00 

Std  Deviation 

0.511 

Functionality  and  ease  of  use  with  respect  to  Secure  HTML  Survey  page  once 
again  received  mixed  results  due  to  a  lack  of  understanding  between  the  HTML  Survey 
page  and  Secure  HTML  Survey  page  (Figure  4.47  and  Table  4.36).  Participants’  response 
to  functionality  and  ease  of  use  to  the  ACSA  newsgroup  page  indicated  a  lack  of  • 
understanding  of  what  the  page  was  design  for  (Figure  4.48  and  Table  4.37).  Several 
responses  to  these  two  questions  were  left  blank  and  only  eleven  participants  indicated  a 
response. 
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Figure  4.47.  Secure  HTML  Functionality 
Table  4.36  Response  Statistics  for  Question  2.30 


|  5HSiiSl  '  -  ||  II  -  m  Value 

Mean 

2.17 

Median 

2.00 

Std  Deviation 

0.752 

Question  2.31:  Functionality  &  Ease  of  ACS  A  newsgroup  page. 


Figure  4.48.  ACSA  Newsgroup  Functionality 
Table  4.37  Response  Statistics  for  Question  2.3 1 


Mean 

1.33 

Median 

1.00 

Std  Deviation 

0.516 
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Two  questions  were  used  for  comparison  between  the  HTML  survey  and  Survey 
Applet.  The  HTML  format  was  perferred  over  the  Survey  Applet  (Figure  4.49  and  Table 
4.38).  The  Survey  Applet  was  less  desired  by  most  the  participants  due  to  flashing  text, 
alignment  while  reading  and  selecting  the  answer,  layout  of  page  and  scrolling  to  read 
each  question  contributed  to  negative  views  (Figure  4.50  and  Table  4.39). 

Question  2.32:  HTML  is  easier  to  use. 


Poor  Acceptable  Good  Very  Good  Excellent 


Figure  4.49.  HTML  Comparison 
Table  4.38  Response  Statistics  for  Question  2.32 


Stalblic 

Value 

Mean 

4.11 

Median 

4.00 

Std  Deviation 

0.582 

Question  2.33:  Applet  easier  to  use. 


z  Disagree  Neither 

Strongly  Disagree 

nor  Agree 


Agree 

Strongly 


Figure  4.50.  Applet  Comparison 
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Table  4.39  Response  Statistics  for  Question  2.33 


'  '  :: ■■  s 

Valae  m 

w  *  *'\  ' 

Mean 

2.56 

Median 

2.00 

Std  Deviation 

0.983 

The  last  question  asked  participants  to  rate  how  satisfied  they  were  with  the 
ACSA  Web  site  (Figure  4.51  and  Table  4.40).  Fifteen  of  the  eighteen  responses  were 
either  satisfied  or  very  satisfied.  From  interview  comments  the  Web  site  was  viewed  to 
be  innovative  and  move  in  the  right  direction  for  collectiong  survey  data.  Several  Safety 
Officers  commented  that  the  Web  site  would  be  welcomed  in  their  squadrons  and  looked 
forward  to  having  it  as  tool  to  produce  mini-surveys  in  the  future. 


Question  2.34:  Overall,  how  satisfied  are  you  with  the  ACSA  Web  site? 


;;;v  r/. ^  |g| 

'  >  : 

r-"  v 

z  Disagree 
Strongly 


Neither 
Disagree 
nor  Agree 


Agree 

Strongly 


Figure  4.51.  Satisfaction  of  ACSA  Web  Site 
Table  4.40  Response  Statistics  for  Question  2.34 


Mean  4.39 

Median  4.50 

Std  Deviation  0.697 
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V.  CONCLUSION  AND  RECOMMENDATIONS 


A.  CONCLUSION 

Knowledge  of  usability  of  Web  based  information  tools  by  aviation  personnel 
holds  value  for  designers  of  Web  pages  and  sites,  as  well  as  the  user  of  those  applications. 
By  knowing  what  to  expect  and  demand,  users  can  shape  the  future  of  these  designs,  and 
by  knowing  what  users  want  and  desire  in  a  Web  site  based  application,  designers  can 
provide  systems  that  are  usable  and  effective.  Several  conclusions  can  be  drawn  from 
survey  results  and  interviews. 

•  Simplicity  is  essential.  Users  preferred  the  simple  HTML  format  to  the 
complex  Applet  format. 

•  Content  is  important.  If  content  is  lacking  the  user  will  look  elsewhere. 

•  Security.  Confidentially  and  anonymity  must  be  intuitive. 

•  Functionality.  Each  navigation  button  must  be  understandable. 

•  Download  time.  Must  understand  end-user  system  requirements. 

•  Page  layout.  Excessive  scrolling  of  pages  must  be  avoided. 

•  Usability  is  the  key  to  success.  Proper  usability  evaluation  will  save  on 
development  and  redesign  cost. 

B.  RECOMMENDATIONS 

The  information  obtained  from  the  literature  and  the  insight  gained  from  the 
surveys  and  observations  lead  to  several  suggestions.  These  concluding  statements 
should  provide  the  stimulus  for  change  supported  by  the  collated  data.  The  results  of  this 
study  and  the  conclusions  drawn  from  the  suggest  that  the  actions  listed  below  will 
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enhance  the  ACSA  Web  site  and  make  it  a  much  easier  application  for  collecting  survey 
data. 

•  Eliminate  the  Survey  Applet  from  the  ACSA  Web  site.  This  will  decrease 
loading  times  and  requirements  for  Java  plug-in  modules. 

•  Rewrite  the  introduction  page  to  include  a  statement  concerning  survey  goals 
and  objectives,  confidentiality  of  responses,  and  anonymity  of  individuals. 

•  Redesign  the  help  page  to  include  a  table  of  contents,  links  and  list  of 
acronyms. 

•  An  introduction  page  needs  to  be  included  before  the  HTML  survey  and 
Survey  Applet.  The  page  should  highlight  survey  objective,  safety  goals  and 
requirement  for  data. 

•  Removes  help  links  from  HTML  survey  and  Survey  Applet.  These  links  only 
send  you  to  the  Help  page  which  do  not  address  any  survey  issues. 

•  Make  either  HTML  survey  or  Secure  HTML  survey  available,  not  both. 
Confusion  exists  between  the  unsecured  and  secure  formats. 

•  Provide  links  to  other  related  Aviation  Web  site. 

C.  FUTURE  RESEARCH 

The  recommendations  offered  are  an  indication  that  future  work  can  be  done  on 
the  ACSA  Web  site.  Research  into  the  saving  a  partially  completed  survey  is  probably 
necessary  in  the  event  an  individual  cannot  complete  the  survey  and  must  return.  Another 
area  of  research  applicable  to  the  ACSA  Web  site  concerns  the  effectiveness  and 
efficiency  of  administering  the  ACSA  survey  electronic  over  a  traditional  paper. 
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A  study  relating  the  effects  of  linked  information  should  be  performed.  This  study 
should  attempt  to  determine  whether  users  prefer  links  to  information  that  may  not  be 
entirely  accurate,  or  no  link  at  all.  It  could  also  explore  the  effects  of  linked  information 
on  the  referring  Web  site. 

The  lack  of  literature  in  this  area  suggests  a  fertile  area  to  be  probed  by  future 
researchers.  Additional  bodies  of  data,  large  enough  to  be  statistically  significant  should 
be  gathered  and  analyzed.  Continued  surveying  with  the  current  surveys,  or  a  similar 
updated  survey  of  this  type,  would  provide  further  insight  into  the  aviation  community  as 
a  user  of  this  system. 
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APPENDIX  A.  GENERAL  WEB  SURVEY 


Note:  Due  to  formatting  differences  this  survey  changed  slightly  in  appearance. 
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GENERAL  WEB  SURVEY 


Please  read  the  following  before  continuing . 

Information  gained  by  this  survey  will  be  used  in  a  thesis  aimed  at  creating 
a  better  Web  interface  design  for  the  Aviation  Command  Safety 
Assessment  (ACSA).  Accurate  information  is  essential  to  developing  an 
accurate  profile  of  indented  users.  Feedback  will  also  be  integrated  into 
the  ACSA  Web  site.  The  information  obtained  will  also  allow  us  to 
profile  the  typical  users  squadron  level.  The  information  obtained  in  this 
survey  is  and  will  remain  anonymous;  no  names  will  be  taken  or 
associated  in  any  way  to  the  information  gained. 


This  survey  should  take  approximately  10  to  15  minutes  to  complete. 


Thank  you  for  taking  the  time  to  participate. 


General  Web  Survey 


Please  take  a  few  minutes  to  complete  this  survey.  Your  specific  answers  will  be  completely 
anonymous,  but  your  views,  in  combination  with  those  of  others,  are  extremely  important. 


1.  How  many  hours  per  day,  on  average  do  you  spend  at  work? 

8-10  hrs . d  10-12  hrs . d  more  than  12  hrs . d 

2.  What  is  your  age? 

Under  25 . □  25  to  27 . □  28  to  30 . □  31  to  33 . □  33  or  older . □ 

3.  What  is  your  sex? 

Male . d  Female . d 

4.  What  is  your  branch  of  service? 

Army . d  Coast  Guard . d  Marine  Corps . d  Navy . d  Other . d 

5.  What  is  your  pay  grade? 

02 . d  03 . d  04 . d  05 . d  Other . d 

6.  I  use  a  computer  at  (please  check  all  that  apply): 

Home . d  Work . d  Mobile . d 

7.  I  use  (please  check  all  that  apply): 

a  PC . d  a  Macintosh .  . □ 

a  Palm  Top . d  Windows  95/98 . d 

MS  Internet  Explorer. . . .d  MS  Office  95/97 . d 

Other  (please  write  in): _ 

8.  What  type  of  activities  do  you  perform  on  computers  (please  check  all  that  apply,  whether  at  work  or  at  home): 


E-mail . 

. □ 

Word  Processing . 

. □ 

Web  Access/Browsing . 

...□ 

Games . 

. □ 

Presentations  Graphics . . 

. □ 

DoD  Applications  (Supply,  Travel).. 

Spread  sheet . 

. . □ 

Financial  Programs . 

. □ 

Other . 

..□ 

a  Notebook .  . □ 

Netscape  Browser . HU 
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General  Web  Survey 


Please  take  a  few  minutes  to  complete  this  survey.  Your  specific  answers  will  be  completely  anonymous,  but  your  views,  in 
combination  with  those  of  others,  are  extremely  important. 


9.  Have  you  ever  completed  a  formal  computer  course? 

Yes . □  No . □ 

10.  How  would  you  rate  your  computer  skills? 

Novice . d  Intermediate . d  Expert . d 

Internet  use: 

11.  In  total,  approximately  how  much  time  per  day,  on  average,  do  you  spend  on  the  Internet  (i.e.  Browsing  or  searching  the 
web)  for  any  purpose? 

0-30  minutes...  . □  30-60  minutes . □  1-3  hours . □  More  than  3  hours . . .  . □ 

12.  Approximately  how  often  do  you  Browse  or  Search  the  Web? 

Several  times  daily . d  Once  daily . d  Less  than  five  times  per  week . d 

Less  than  ten  times  per  month . d  Never .  . □ 

13.  When  you  get  to  a  Web  site,  how  long  do  you  stay  there? 

0-2  minutes . d  2-10  minutes . d  10-30  minutes . d  More  than  30  minutes . d 

14.  When  you  are  finished  looking  at  a  Web  site,  do  you  (rank  in  order,  with  1  being  most  used  action): 


Follow  a  query-retrieved  suggestion  (i.e  go  back  to  a  search  engine  and  follow  a  link  from  there) .  . □ 

Jump  to  another  Web  site  without  using  a  link .  . □ 

Follow  a  link  to  another  Web  site .  . □ 

Close  the  browser .  . □ 

N/A,  I  do  no  access  the  Web . d] 


15.  Regarding  the  graphic  user  interface  used  on  Web  sites  you  visit,  what  do  you  find  most  detracting? 

(Rank  in  order,  with  1  being  most  detracting) 

Long  download  times .  . □  Counter  intuitive  links . □  Non-standard  link  colors . □ 

Long  scrolling  pages .  . □  Lack  of  navigation  support...  □  Poor  color  schemes . □ 

Scrolling  text,  marquees,  ads,  constantly  running  animation .  . □ 
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General  Web  Survey 


The  following  questions  are  for  analytic  purposes  only.  They  will  not  be  used  to  try  to  identify  any  individual. 
However,  if  you  feel  uncomfortable  about  answering  any  of  them  do  not  do  so. 


Disagree  Neither  Disagree  Agree 

Strongly  nor  Agree  Strongly 


16.  I  view  Web  site  address  as  complex  and  hard  to  use . 1 . 2 . 3 . 4 . 5 

17.  Computers  make  me  more  productive . 1 . 2 . 3 . 4 . 5 

18.  Computer  increase  my  workload  . 1 . 2 . 3 . 4 . 5 

19.  Computer  reduce  paperwork . 1 . 2 . 3 . 4 . 5 

20.  List  3  Web  sites  you  like  and  give  reasons(s)  you  like  each  one  (you  do  not  have  to  use  the  exact  address) 

NOTE:  If  you  can  not  think  of  3  sites,  please  list  3  features  you  like  about  Web  sites. 


21.  List  3  Web  sites  you  do  NOT  like  and  give  the  reason(s)  for  each  one  (you  do  not  have  to  use  the  exact  address) 
NOTE:  If  you  can  not  think  of  3  sites,  please  list  3  features  you  do  NOT  like  about  Web  sites. 
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APPENDIX  B.  LOG  ON  PROCEDURES 
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Welcome  to  the  Aviation  Command  Safety  Assessment  (ACSA) 


We  would  like  to  thank  you  for  participating  in  this  usability  analysis.  Our  goal  is  to  learn  how  we  can  improve  this  web  site 
for  the  fleet.  Your  comments  will  assist  in  making  the  ACSA  another  valuable  resource  for  naval  aviation.  The  survey 
consists  of  nine  biographical  questions  and  61  survey  questions.  We  assume  you  are  familiar  with  basic  browser/web  site 
navigation.  Due  to  the  length  of  the  survey,  this  analysis/interview  should  take  roughly  30-45  minutes. 


I.  Completing  the  Usability  Analysis  HTML  Format 

1 .  Open  your  browser  of  choice  to  begin,  either  Netscape  or  Internet  Explorer. 

2.  Enter  the  following  Internet  address  httv://svUfire.avsafetv.nvs.navv.mil  to  access  the  ACSA  home  page. 

3.  After  accessing  the  ACSA  home  page,  you  will  be  asked  to  enter  a  User  Name  and  Password.  Enter  the  following: 

Use  name:  Don’t 
Password:  XXXX 

The  ACSA  home  page  provides  general  information  for  the  user.  Please  take  a  moment  to  read  this  page  before  proceeding. 

4.  Select  HTML  Survey  to  begin. 

5.  You  should  now  be  viewing  a  survey  form  containing  Biographical  Information  and  Survey  Questionnaire  sections. 
For  purposes  of  this  evaluation  answer  the  Biographical  Information  questions  with  your  current  personal  data. 

6.  Answer  all  Survey  Questions  from  the  perspective  of  your  last  flying  command.  Use  the  drop-down  list-boxes  to 
select  your  appropriate  entry  for  each  question. 


II.  Submitting  your  Survey 

1 .  Click  the  SUBMIT  button  at  the  bottom  of  the  page  once.  This  will  open  a  dialog  box  requesting  your  squadron 
number. 

Note:  If  you  made  an  error  completing  the  form,  or  forgot  to  answer  a  question  then  the  program  will  alert  you  to  complete 
the  appropriate  question.  After  completing  this  question  re-click  the  SUMBIT  button 

2.  Type  your  squadron  number  for  the  squadron  number  and  then  click  OK.  This  will  open  another  identical  dialog  box 

requesting  your  survey  number.  Your  survey  number  is _ 

3.  Type  your  assigned  survey  number  and  click  OK.  If  your  submission  entries  were  correct,  you  will  see  the  Thank  You 
page,  or  else  you  will  see  a  warning  page. 
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III.  Viewing  Results 


The  ACSA  program  provides  commanders  in  the  field  the  opportunity  to  query  the  ACSA  database.  The  squadron  ASO  and 
CO  will  have  a  special  password  to  access  this  page. 

1 .  Select  Results  Applet  to  view  results.  You  will  get  a  warning  that  Authorization  failed.  Enter  the  following: 

User  Name:  Commander 

Password:  XXXXXXXX 

2.  To  view  results: 

1 .  Select  a  question  from  the  drop-down  box. 

2.  Click  SUBMIT. 

Queries  can  be  limited  by  selecting  additional  query  parameters  from  the  three  remaining  drop-down  boxes.  Note  the  first 
query  takes  several  (10  -  151  seconds  to  process  over  the  network,  but  subsequent  queries  are  much  faster. 


Thank  you  for  taking  the  time  to  complete  this  survey. 
We  are  interested  in  your  comments! 
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APPENDIX  C. 


AVIATION  COMMAND  SAFETY  ASSESSMENT  (ACSA)  USABILITY  SURVEY 


Note:  Due  to  formatting  differences  this  survey  changed  slightly  in  appearance. 
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ACSA  USABILITY  SURVEY 


Please  read  the  following  before  continuing . 

Information  gained  by  this  survey  will  be  used  in  a  thesis  aimed  at  creating  a  better  Web 
interface  design  for  the  Aviation  Command  Safety  Assessment  (ACSA).  Accurate  information 
is  essential  to  developing  an  accurate  profile  of  indented  users.  Feedback  will  also  be  integrated 
into  the  ACSA  Web  site.  The  information  obtained  will  also  allow  us  to  profile  the  typical  users 
squadron  level.  The  information  obtained  in  this  survey  is  and  will  remain  anonymous;  no 
names  will  be  taken  or  associated  in  any  way  to  the  information  gained. 


This  survey  should  take  approximately  20  to  30  minutes  to  complete. 


Thank  you  for  taking  the  time  to  participate 
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Aviation  Command  Safety  Assessment  (ACSA)  Usability  Survey 


Please  take  a  few  minutes  to  complete  this  survey.  Your  specific  answers  will  be  completely  anonymous,  but  your  views,  in 
combination  with  those  of  others,  are  extremely  important. 


Section  I  -  General 


Regarding  the  ACSA  Web  site,  please  respond  to  each  of  the  following: 


1. 

Information  on  the  site  is  relevant . 

Disagree 

Strongly 

. i . 

. 2. 

Neither  Disagree 
nor  Agree 

. 3 . 

..4 . 

Agree 

Strongly 

. 5 

2. 

Information  on  the  site  is  accurate . 

. 1 . 

. 2. 

. 3 . 

..4 . 

. 5 

3. 

Information  on  the  site  is  complete . 

. i . 

. 2. 

. 3 . 

..4 .... 

. 5 

4. 

Information  on  the  site  is  current . 

. i . 

. 2.. 

. 3 . 

..4.... 

. 5 

5. 

Information  on  the  site  meets  my  needs . 

. 1 . 

. 2.. 

. 3 . 

..4.... 

. 5 

6. 

Information  on  the  site  is  intuitive . 

. 1 . 

. 2.. 

. 3 . 

..4.... 

. 5 

7. 

Information  on  the  site  is  easily  readable . 

. i . 

. 2., 

. 3 . 

..4.... 

. 5 

8. 

The  site  address  is  too  complex . 

. i . 

. 2., 

. 3 . 

..4.... 

. 5 

9. 

The  amount  of  page  scrolling  is  acceptable . 

. i . 

. 2., 

. 3 . 

..4 .... 

. 5 

10. 

The  link  colors  are  acceptable . 

. 1 . 

. 2.. 

. 3 . 

..4.... 

. 5 

11. 

The  colors  used  throughout  the  site  are  acceptable . 

. 1 . 

. 2.. 

. 3 . 

..4 .... 

. 5 

12. 

The  amount  of  graphics  is  acceptable . 

. i . 

. 2.. 

. 3 . 

..4.... 

. 5 

13. 

I  would  like  to  see  more  graphics . 

. 1 . . 

. 2.. 

. 3 . 

..4.... 

. 5 

14. 

Download  times  are  acceptable  (HTML  format)  . 

. 1 . 

. 2.. 

. 3 . 

..4.... 

. 5 

15. 

Download  times  are  acceptable  (Java  applet)  . 

. 1 . 

. 2.. 

. 3 . 

..4.... 

. 5 

16. 

Navigation  support  is  acceptable:  . 

. 1 . 

. 2.. 

. 3 . 

..4 .... 

. 5 

17. 

Navigation  through  the  site  is  intuitive . 

. i . 

. 2.. 

. 3 . 

..4 . 

. 5 
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Aviation  Command  Safety  Assessment  (ACS A)  Usability  Survey 


Please  take  a  few  minutes  to  complete  this  survey.  Your  specific  answers  will  be  completely  anonymous,  but  your  views,  in 
combination  with  those  of  others,  are  extremely  important. 


Please  answer  the  next  set  of  questions  regarding  how  you  feel  about  each  of  the  following  specific  matters? 

Go  to  the  ACSA  Web  site  located  at  http://spitfire.avsafetv.nps.navv.mil 

Web  site  design/Overall  presentation  (Please  circle  one  number  for  each  statement) 


18. 

19. 

20. 
21. 
22. 

23. 

24. 


Main  page . 

HTML  Survey  page . 

Survey  Applet  page . 

Results  Applet  page . 

Help  page . 

Secure  HTML  Survey  page 
ACSA  newsgroup  page . 


Poor  Acceptable  Good  Very  Good  Excellent 
...1 . 2 . 3 . 4 . 5 

...1 . 2 . 3 . 4 . 5 

...1 . 2 . 3 . 4 . 5 

...1 . 2 . 3 . 4 . 5 

...1 . 2 . 3 . 4 . 5 

..1 . 2 . 3 . 4 . 5 

..1 . 2 . 3 . 4 . 5 


Functionality/Ease  of  use  (Please  circle  one  number  for  each  statement) 


Poor  Acceptable  Good  Very  Good  Excellent 


25.  Instructions  on  Main  page .. 

26.  HTML  Survey  page . 

27.  Survey  Applet  page . 

28.  Results  Applet  page . 

29.  Help  page . 

30.  Secure  HTML  Survey  page 

3 1 .  ACSA  newsgroup  page . 


1 . 2 . 3 . 4 . 5 

1 . 2 . 3 . 4 . 5 

1 . 2 . 3 . .4 . 5 

1 . 2 . 3 . 4 . 5 

1 . 2 . 3 . 4 . 5 

1 . 2 . 3 . 4 . 5 

1 . 2 . 3 . 4 . 5 
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1 


Aviation  Command  Safety  Assessment  (ACSA)  Usability  Survey 


Please  take  a  few  minutes  to  complete  this  survey.  Your  specific  answers  will  be  completely  anonymous,  but  your  views,  in 
combination  with  those  of  others,  are  extremely  important. 


In  comparison  to  the  HTML  Survey  and  Survey  Applet 

32.  HTML  is  easier  to  use . 


Disagree  Neither  Disagree 

Strongly  nor  Agree 

. 1 . 2 . 3 . 4 


Agree 

Strongly 

. 5 


33.  Applet  easier  to  use 


2 . 3 . 4 . 5 


34.  Overall,  how  satisfied  are  you  with  ACSA  Web  site?  (Please  circle  one  number) 


Very 

Neither  Satisfied 

Very 

Dissatisfied 

Dissatisfied 

nor  Dissatisfied 

Satisfied 

Satisfied 

i 

2 

3 

4 

5 
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APPENDIX  D.  INTERVIEW  QUESTIONS  AND  SIGNIFICANT  RESPONSES 


Question:  Did  you  find  the  portion  of  the  ACSA  that  you  used  to  be  an  easy  tool  for 
completing  the  survey? 

Responses  (HC-3,  HC-11  and  HSL-45): 

•  Yes,  survey  (HTML)  was  easy  to  access  and  easy  to  complete. 

•  Yes,  very  logical  and  something  that  will  add  value  to  the  fleet. 

•  Yes,  great  Web  site. 

•  Yes,  but  why  are  there  two  HTML  formats? 

•  Yes,  quick  to  use  and  easy  to  submit. 

•  Yes,  I  thought  it  was  straight  forward  and  intuitive. 

•  Yes,  super  graphics  and  nice  layout. 

•  No,  Java  plug-in  take  to  long  to  load  and  download  times  are  excessive. 

•  No,  what  is  a  ACSA  newsgroup. 

•  No,  Survey  Applet  won’t  run  right,  but  HTML  format  is  veiy  nice. 

Responses  (HC-2,  VAW-120  and  VAW-123) 

•  Yes,  way  to  go  -  look  forward  to  it  getting  to  the  squadrons. 

•  Yes,  will  make  my  job  easier  as  Safety  Officer. 

•  Yes,  excellent  tool  to  have  in  my  bag  of  tricks. 

•  Yes,  CO’s  will  fun  it. 

•  Yes,  but  is  my  answer’s  secure. 

•  Yes,  but  why  the  different  formats  (HTML,  Secure  HTML,  Survey  Applet)? 

•  Yes,  nice  work  on  Web  site  and  good  graphics. 

•  Yes,  but  what  happens  if  there  is  a  power  outage  while  taking  the  survey? 

•  No,  Survey  Applets  take  too  long  to  load. 

•  No,  Help  is  very  helpful. 

•  No,  too  much  scrolling  -  can’t  reset  my  screen. 

•  No,  downloading  plug-in  is  a  pain  in  my  %#*. 


85 


Question:  Was  there  anything  that  you  found  particularly  difficult  to  do  or 
understand? 

Responses  (HC-3,  HC-11  and  HSL-45): 

•  Not  really 

•  No,  easy  to  use  survey 

•  No,  but  I  couldn’t  get  Survey  Applet  to  work  at  first. 

•  Yes,  what  is  ASCA  newsgroups. 

•  Yes,  difference  between  Secure  HTML  and  HTML  survey. 

•  Yes,  help  is  not  a  help  function. 

•  Yes,  Results  applet  takes  to  long  to  download 

•  Yes,  why  do  I  have  navigation  buttons  to  items  I  will  not  access  in  the  future 
(i.e.  Results  applet,  ACSA  newsgroup)? 

•  Yes,  how  is  the  data  secured? 

•  Yes,  what  protects  me  from  someone  finding  out  what  my  answers  are. 

Responses  (HC-2,  YAW-120  and  YAW-123) 

•  No,  but  the  Main  page  instructions  are  too  technical. 

•  No,  but  the  help  menu  is  not  index  to  find  anything. 

•  Yes,  what  is  ASCA  newsgroup  and  who  uses  it. 

•  Yes,  Java  plug-in  would  not  load  and  finally  took  30  minutes. 

•  Yes,  what  protects  me  from  someone  knowing  my  responses? 

•  Yes,  why  is  there  a  Secure  HTML  survey  is  it  better  than  HTML  survey? 

Question:  If  you  could  change  one  thing  about  the  ACSA  Web  site  to  make  it  easier 
to  use,  what  would  it  be? 

Responses  (HC-3,  HC-11  and  HSL-45): 

•  Delete  the  Survey  Applet,  take  too  long  to  load. 

•  Delete  flashing  text  and  waving  text  in  Survey  Applet. 

•  Make  the  Help  function  better 
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•  Expand  the  list  of  terms. 

•  Too  much  scrolling  to  read  information. 

•  No,  easy  to  use  survey 

•  Survey  Applet  does  not  have  “select  from  the  following”. 

•  Add  Likert  scale  at  the  top  of  each  survey 

•  Flashing  titles  are  annoying 

•  Options  to  go  back  and  change  selection  “Need  back  button”. 

•  Small  type  size  and  too  scrolling. 


Responses  (HC-2,  VAW-120  and  VAW-123) 

•  Main  page  scrolling  too  fast  to  read. 

•  Make  statements  that  answers  are  will  be  confidential  in  open  introduction. 

•  State  purpose  of  survey. 

•  Applet  page  should  have  note  to  continue  with  “Tab  Questions”. 

•  Need  Likert  scale  at  top  of  page. 

•  Main  page  -  need  statement  about  “your  answers  will  remain  confidential,  no 
attempts  will  be  made  to  identify  you  or  your  organization”. 

•  Help  need  some  type  of  index. 

•  Result  page  -  need  to  select  both  coast  at  once. 

•  Delete  loading  Java  plug-in,  what  a  waste  of  time. 

•  Either  keep  Secure  HTML  survey  or  HTML  survey,  but  not  both. 

•  One  need  one  type  of  survey  -  HTML  works  just  fine. 

•  Too  much  scrolling  -  make  page  that  fix  monitor. 


Question:  Are  there  any  features  you  would  like  to  see  added  to  the  ACSA  Web 
site? 

Responses  (HC-3,  HC-11  and  HSL-45): 

•  Links  to  other  aviation  safety  issues. 

•  Access  to  Results  page  by  Safety  Officer  and  ASO’s. 

•  A  better  help  feature  not  just  a  list  of  questions  answered. 
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Responses  (HC-2,  VAW-120  and  VAW-123) 


•  Larger  text  size. 

•  Looks  great  the  way  it  is. 

•  I  like  the  lay  out  but  scrolling  is  a  problems. 
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APPENDIX  E.  SPREADSHEET  DATA  FOR  THE  GENERAL  WEB  SURVEY 
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GENERAL  WEB  SURVEY  DATA 
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IE  =  MS  Internet  Explorer  FIN  =  Financial  Programs 
OFF95  =  MS  Office  95/97 


GENERAL  WEB  SURVEY  DATA 


Mean _ 2.36  1.86  3.57  1.79 

Median _ 2m _ 2JOO _ 3.50  1.00 

Standard  Deviation  0.928783  0.6629935  1.157868  1.050902 


GENERAL  WEB  SURVEY  DATA 


If 

O 

0 

« 

£ 

| 

5> 

■o 

c 

CO 

o 


o 

c 

o 

"D 

3 

O 

> 

0) 

'</> 

.Q 

I 

CO 

#■» 

CO 


IT 

o 

<0 

CO 

0) 

k. 

0) 

> 

'5> 

■o 

c 

CO 

I 

§ 

>» 

a> 

'3 

sx 

0) 

5 

CO 

o5 


c 

CO 


CO 

x: 

0  -Q 

>s  c 

$ 

o 


05 

c 


<0 

0 

c 

’O) 

c 

LU 

JZ 

£ 

CO 

0 

CD 


*2 

C0 

x: 

a> 

> 


5  2 

o 
co 


^  J2 

S-E 
.  c 

"2  © 

2  2 
lu  m 


q 

o 

c 

_c0 

00 

X) 

3 

CO 

o 

c 

3 

X2 

0 

CO 

E 

2 

CL 


0 

> 

'■*—> 

§ 

0 

•4—* 

c 


0  -7 

O  ^ 
XI  1 
CL  0 

2  2 
0  *0 

0  © 

3  5 
O  0 
-■ti  c 

3  — 
2  § 
0  5 


J>s 

T3 

C 

0 


0 
0 
3 

O 

c 

•g  ffi 

2  j= 

*5  o 
or  O 

-jT 

1- 

O  0  5 

E  E  S 

XI  £  CO 
O  _  CD 

S  -g  «D 

°  c  ^ 
O  C  T5 

T  o  LU 


"O 

©  0 
0  XI 
a  -3 

:=  o 
Q-  0 
£  -9 

g 

o 
o 


Si 


c 

0 

■8 

1 

0 

“D 

"c 

0 

0 

O 

“D 

xf 

O) 

p  f 

co 


O  2 
CE  • 

O0 
0 

ll  C 
1 —  “ 


H  <  m 


c  0  O 

•  c  c 
X  C/3  H) 
©  CE  ? 
§  U-1  In 
co  Q-  co 
J0  z>  J0 
LL  CO  LL 


0 

05 

0 

a. 

0 

£ 

77  -o  £ 
^  co  o 

,5  2  0 

2  o  -o 

or  § 
0  .2 
^  TO 

0  E  co 

£  0  -c 


2  © 


0 

0  ^  0 
05  ><  05 
0  0  0 
0-  |-  CL 


0 

E 

o 

c 

0 

c 

■c 

0 

Q_ 

O 

c 


2  o 

r-  **— 


O 

C 

■D 

O 

O 

05 


05 

n  o 

H- 

0  .£ 


0 

0 

0 

O 

o 

0 


o 

CD 

h— 

o 

X 


0 

0 

i— 

O) 

I 

0 

■c 

o 

Q_ 

CD 


CL 
CD 
CD  LU 


O 

c 

0 

> 

*0  O 
C  r- 
0  .£ 
x:  0 

2  > 
a_  0 

o  J 

°- 2 
Q. 

•i  E 

a-  8 

i  _ 

J- 

o  o 

o  3 
T3  cr 

0  I 
0  — 
*3  0 

2  i 
©  0 
cl  x: 

2  O 

0“  0 

^  2 
0  0 
x  0 

cL^ 
0  0 
c  xi 
CD  H 


o 

0  c 
5  X 
0  3 


2  £ 

0  *> 

0  > 

o  &  © 
O  m  2 

8  S.  § 

o  ?  “ 
O  t  0 


0 

0 


ST 

0 

0 

"O 

c 

0 


£  8 

•£  § 
0  ^ 
tZ  0 

&g 

0  .£ 
x: 


o> 

* 

Jsi 

o 

0 

Q 


o  a 
-c  c 


E 

o 

u.  ■  9 
Oh  t 
X  LU  o 

5  o  § 

gl5 

TJ  3  CD 
N  Z  LU 


>s 

0 

0 

0 

d) 

c 

2 


0  c  ©  O  2 


3 
0 
0 
0 

2  -O 

©3  o 

0  • 
c  r- 

9  0 
‘o  2 
Q  Q 


0 

0 

o 

3 

O’ 

o 

2 

co 


3 

o 

T3 

C 


0 

> 

0 

0 


.  0 
0  £ 
O)  0 

s? 
^  0 
—  0 
0  C 


0 
3 

m 


0  0 
®  •§>•- 
1S| 

05-§  O 
*  «  ® 

^  £ 
_0  0 

'5_  > 

o  2 


LU  Q  < 


■O 

0 

^  ^0 
0  0 
X  •- 
0  2 

!l 

0 

^  s> 

0  3 
C 

0  o 

0  C 
X3  *— 

£1 

8  r 

1 1 

|S 

f  8 

g  J 

IS 


0 

0 

*0 

T3 

0 

2 

0 


0 

C 

O 

*3 

2 

c 

o 

’3 

O’ 


X2 

I 

0 


c 

o 

0 

tr 

o 

CL 

0 

0 

0 

X} 


3 

0 


1  « 

2  a) 


o 
o 

0 
0 

"E 
o 

18 

<  < 


©  X 
£  0 

O  m 
CL 

CO  -2 


a.  i 

LU  O 


O  “ 
a>  -t:  c 

i|i 

8>—  » 
0)-S  2 

c£.= 
o  05  1 

O  '  JD 

■o  O  § 

figl 

^08 


--j  0 

O  2 
o  o 

05  3 

.  cr 

Ex 

o 

O  O  © 


2® 
c 

2  0 

0  o 
.9  3  — 

CL  O-LU 


tr 

o 

CL 

CD 

Z 

0- 

CD 


92 


ABC.com  -  nice  layout 
KellyBlueBook  -  quick,  easy  to  read 
Flight  Info  -  easy  to  use  and  fast 
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ACSA  SURVEY  DATA 
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